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Abstract 


Let il C (N > 3) be an open domain (may be unbounded) with 0 € dft and dft 
be of at 0 with the negative mean curvature H{0). By using variational methods, 
we consider the following elliptic systems involving multiple Hardy-Sobolev critical 
exponents, 


-Au + A*^ 


— Ai 

- Ml 


|xpi 

\v\^ 

Ixpl 


{u,v) € Dl’^{n) X £>o’^(0), 


= Ai^|u|“ ^u\v\^ 


in n, 
in n, 


where 0 < cto, po, S 2 < 2, si G (0, 2); the parameters A* 7 ^ 0, p* ^ 0, Ai > 0, /ii > 
0, A/r > 0; a,/3 > 1 satisfying a + /3 < 2*(s2). Here, 2*(s) := is the critical 

Hardy-Sobolev exponent. We obtain the existence and nonexistence of ground state 
solution under different specific assumptions. As the by-product, we study 


[ Au + A^ + t^ 
\ u{x) > 0 in fl, 

[ u{x) = 0 on i90, 




= 0 


in H, 


( 0 . 1 ) 


we also obtain the existence and nonexistence of solution under different hypotheses. 
In particular, we give a partial answers to a generalized open problem proposed by 
Y. Y. Li and C. S. Lin (ARMA, 2012). Around the above two types of equation or 
systems, we systematically study the elliptic equations which have multiple singular 
terms and are defined on any open domain. We establish some fundamental results. 


Key words: Elliptic system. Ground state, Hardy-Sobolev exponent. 
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1 Introduction 


Let fl C {N > 3) be an open domain with 0 € dfl and dfl be of at 0 with the 
negative mean curvature H (0). We study the following nonlinear elliptic systems 


-Au + X* - Ai 


|u|2*('*l>-2i 

Ixpl 

-Av + u*— _til 

{u,v) e iLo(fl) X iLo(n), 


= A 


= M 


\x\^2 


\u\^ 


u\ \v 




- 2 , 


in 17, 
in 17, 


( 1 . 1 ) 


Note that the parameters A*, /r*, Ai, /ri, A, /r may different from each other. Hence, the 
system has no variational structure. 

The interest in nonlinear Schrodinger systems is motivated by applications to non¬ 
linear optics,plasma physics, condensed matter physics, etc. For example, the coupled 
nonlinear Schrodinger systems arise in the description of several physical phenomena 
such as the propagation of pulses in birefringent optical fibers and Kerr-like photore- 
fractive media, see [2, 9, 16, 26, 27, 33], etc. The problem comes from the physical 
phenomenon with a clear practical significance. Research on solutions under different 
situations, not only correspond to different Physical interpretation, but also has a pure 
mathematical theoretical significance, with a high scientific value. Hence, the cou¬ 
pled nonlinear Schrodinger systems are widely studied in recently years, we refer the 
readers to [1, 3, 22, 25, 29, 36, 37] and the references therein. 

^ \u\" 


For any s € [0, 2], we define 


Hip = 


-dx. We also use the notations 


/n FI' 


U D = 


||it||o,p. The Hardy-Sobolev inequality [5, 6, 13] asserts that Dq 


t,N\ 


L2-is) 




dx 


1 an embedding for s S [0,2]. For a general open domain f7, there 


exists a positive constant C{s, 17) such that 


[ |Vu|2>C(s,17)f / 
7n \Jn 


Define ^si (1^) is defined as 


| 2 *(s) 


-dx 


2 ^ 




fisi = inf < 


fa iVupdx 


; u e } ■ 


[Ja 


( 1 . 2 ) 


Consider the case of 17 = K])., it is well known that the extremal function of /isj 
is parallel to the ground state solution of the following problem: 


' A k|2*(“i)-2r 

M = 0 on 


in. 




(1.3) 


■+ ■ 


We note that the existence of ground state solution of (1.3) for 0 < si < 2 is solved by 
Ghoussoub and Robert [12]. They also gave out some properties about the regularity. 


3 










symmetry and decay estimates. And the instanton U(x) := C (k + \x'^ ^ for 

0 < S2 < 2 is a ground state solution to (1.4) below (see [20] and [32]): 


Au + 




\ x \‘2 


u > 0 in ! 


= 0 

and 


in 

u ^ 0 as |x| —> +00. 


(1.4) 


That 0 G on and dfl is smooth at 0 has become a hot research topic in recent years as 
the curvature at 0 plays an important role, see [8, 11, 12, 14] .etc. When 0 G dfl with 
H{0) < 0, Hsi (^2) < fisi (K+) was proved in [11] for A > 3 and [21] for A > 4. It is 
standard to apply the blow-up analysis to obtain that ^si (n) can be achieved by some 
positive u G Aq (n) (see[l 1, Corollary 3.2]), which is a ground state solution of 




—Art =- 1 la. 

Ixpi 

u = 0 on on. 


in n, 


and the ground state value equals 



Msi (^) 


N-si 

2-si 


Although (1.1) has no variational structure, for k G M such that kX > 0, we can 
take ti >0,^2 >0 properly such that 



Denote u = tiu,v = t 2 V, then a direct calculation show that {u,v) is a solution to 
(1.1) if and only if (u, v) is a solution to the following problem: 




\x\^0 -L fI ^ 

* V ~ 


—Au ■ 


(u,v) G (n) X D^' (n), 


Kaj^\u\°‘ ^u\v\^ 


in n, 
in n, 


(1.5) 


where Ai = Xit\ ^ ^ Hence, without loss of generality, we will 

study the following problem: 




—Au 
—Au' 


A* 




|x|'^o 


-A 


(u,u) G Ao’^(n) X £>p’^(n), 


i«i 


2*(n)-2,, 


|xpi 
Gi)- 2, 




in n, 
in n, 


( 1 . 6 ) 


where k is a parameter. 

There seems to has no article involves the system case with 0 G dVl. Hence, in 
present paper, our aim is to establish related results for the elliptic systems on the do¬ 
main with 0 G do,. There also seems to have no article studies the unbounded domain 

which is not a limit domain (i.e., fl A lim —, where — := 1 a; G K : te G H [). We 

^ t^o t t ^ ’ 

note that the limit equation of (1.6) has been studied in [40, 41]. 
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We are interested in the existence of nontrivial ground state solutions, hence we 
need a well study in a single equation, especially for the unbounded domain case. We 
obtain the following results: 


Theorem 1.1. Assume that A* > 0, C is domain such that |n| < cxd, 0 G dfi 
and the boundary dfl is at 0 with the mean cun’ature H{0) < 0 . Wfe also assume 
that 0 < Si < 1 or 1 < Si < 2 with A* small enough. Then problem 

-Au + A 1 ^ - A - uiQ. 

u{x) = 0 on dft, 


possesses a positive ground state solution u. 

Theorem 1.2. Let fl C R^ be a domain such that |0| < oo, 0 G dfl and the 
boundary dVl is at 0 with the mean curvature H (0) < 0. We also assume that 
—Ai,si(f^) < A* < 0 and 0 < si < 2, then problem (1.7) possesses a positive ground 
state solution u. 


Theorem 1.3. Suppose that Ll C R^ is a open domain with 0 G dfl and dLl is 
at 0, 77(0) < 0. We also assume that 

rA> 0 ,l<p< min{ 2 *(s 2 ) — 1 , 2 *(si) — 1 } if |n| = oo; 


eitherX < 0,1 < p < 2 *(si) — 1 or 
A > 0,1 < p < min{ 2 *(s 2 ) — 1,2*(si) — 1} if |n| < oo. 


Moreover, if X > 0, we require that p < 
Then problem 


-Au + Xj^ = 
u(x) > 0 in n, 
u(x) = 0 on dfl, 
possesses a positive ground state solution. 


or p = 
•(02)-l 


with |A| small enough. 


in n, 


( 1 . 8 ) 


As a product, basing on some lemmas established in present paper for unbounded 
domain, we can also give a partial answers to Li-Lin’s open problem (see [19, Remark 
1.2]) in the unbounded domain case, see Theorem 3.2 and Theorem 3.3 in section 3, 
which are parallel to [7, Theorem 1.5 and 1.6] which are studied in bounded domain. 


We shall prove that when |fl| < oo and 0 < ct < 2, —Au = X- —it G Dq (fl) 

Fr 

possesses a sequence of eigenvalues such that 

0 F Xi^fj X2,a ^ ^ Afc,cr A A/c-i-i^fj A ‘ ‘ ■ 


Define E := n O 


dx 


andF := Dl'^{n)nL^ {Q 


dx 


\xfo 


) . Then 


for some proper range of A* and p*, we can define the following norms: 


u F := 


|Vitp + A*- 


dx 


F F := 


|Vi;( 


dx 
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Our framework space for the system case is ^ := E x F. 

Collect the results about a single equation, we have that equation 


-Am + A* 
u{x) = 0 


Ixl-^o 




|xpi 


in n 
on 90, 


(1.9) 


possesses a positive ground state solution when the following condition {Hi) is satis¬ 
fied: 


(iJi):one of the following holds: 

(a.l) |0| < cxD, — Ai^cro(ll) < A* < 0, 0 < tTo < 2 (see Theorem 1.2). 

(a.2) |0| < oo, A* > 0, 0 < (To < 1 (see Theorem 1.1). 

(a.3) |0| < cxD, A* > 0 small enough, 1 < (Tq < 2 (see Theorem 1.1). 

(a.4) |0| = (X), A* > 0, 0 < (To < 1 (see Theorem 1.3). 

(a.5) |0| = oo, A* > 0 small enough,! < rro < 2 (see Theorem 1.3). 


Let u = u, then m is a ground state solution of 

f _ Af, \* u — |m|^ • Q 

I Am- f A 1^1*1 inSZ, 

[{((a:) =0 on dfl. 


( 1 . 10 ) 


Similarly, equation 


—Am ■ 




1^1 




/t* ^ 0 , M = 0 on dfl 


|a:p 


in O 


( 1 . 11 ) 


possesses a positive ground state solution when the following condition {H 2 ) is satis¬ 
fied: 

{H 2 )‘.one of the following holds: 

(6.1) |r2| < 00 , —Xi^noi^) < /r* < 0, 0 < po < 2 (see Theorem 1.2). 

(6.2) |n| < 00 , /r* > 0, 0 < po < 1 (see Theorem 1.1). 

(6.3) |n| < 00 , /r* > 0 small enough, 1 < po < 2 (see Theorem 1.1). 

(6.4) |n| = cx), /i* > 0,0 < po < 1 (see Theorem 1.3). 

(6.5) |n| = 00 , /r* > 0 small enough,! < po < 2 (see Theorem 1.3). 


Define 


771 := inf — 

.eF\{0} 


II 


n 1x1^2 


dx 


( 1 . 12 ) 
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where 0 < U G E is a positive ground state solution of (1.9). Similarly define 


\E 


r \v\y- 
Jn |a ;|*2 


dx 


where 0 < F € F is a positive ground state solution of ( 1 . 11 ). 
We denote 


(A*) either o-Q < S 2 or 


0 < S 2 < CTo, 

9 j_^ O'0—S2 

0-0 N-2 


< Qi + , 


(1.13) 


(A*J either ryo < S 2 or 


0 < S 2 < ?7o, 


4 V0 — S2 

770 N—2 


< a + /3 


Here comes our first main result for the system case with a + /3 < 2*(s2)- 


Theorem 1.4. (The case of a + /3 < 2*( 32 )) Assume {Hi), {H 2 ),k > 0,1 < a, 1 < 
/3, a + /? < 2 *(s2 ), and !/|H| = 00 , we assume further that (^^ 0 ) (^^ 0 ) V 

{3 < 2 or P = 2 with k > fji, and if a < 2 or a = 2 with k > 7 ) 2 , then problem 


\x\^o 1^1 ^ 

-Aw + u*-^ _„ 

(u, v) € Si, 


= kPt^\u\°‘\v\^-^v 


in H, 
in n, 


(1.14) 


possesses a nontrivial ground state solution {u, v). 

Remark 1.1. When |fl| = ooanda+P < 2 *(s 2 ). the assumptions of {A’)^^) and {A*^) 

play an crucial role to guarantee that is well defined for all {u, v) G ^ (see 

Proposition 2.2). 

To study the critical couple case, for the technical reasons, we need the following 
assumptions (ffi) and {H 2 ), which are stronger than (Hi) and {H 2 )'. 

(iJi):one of the following holds; 

(d.l) |H| < 00 , —Ai,cto(A1) < a* < 0,0 < (To < 2. 

(a. 2 ) |fl| < cx),A* > 0,0 < (To < 1. 

(d.3) |H| = 00 , A* > 0 small enough and (To = 1. 

{H 2 ):one of the following holds; 

( 6 . 1 ) |H| < 00 , —Xi,r]o (^) < /r* < 0 , 0 < po < 2 . 

( 6 . 2 ) |f 2 | < 00 , /r* > 0 , 0 < po < 1 - 

(6.3) |H| = cxD, p* > 0 small enough and rjo = 1. 
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Remark 1.2. When {Hi) and {H 2 ) hold, it is easy to see that {Hi) and {H 2 ) are 
satisfied. 

Theorem 1.5. (The case of a + ^ = 2*{$ 2 )) Assume {Hi), {H 2 ), k > 0,1 < a, 1 < 
fi,a + = 2* { 32 ). Especially, when Si = S 2 = s G (0,2), we assume further that 

one of the following holds: 

\b = 2 

(a.l) \> p.,1 < j3 <2or I rx -i ’’ 

I K > maxjA, 771 1 

(a.2) A = 77 , minja,/3} < 2 or < min{a, ,8} = 2, K > max{A, 771 } 


(a.3) \ < p,,l < a < 2 or 


a = 2 

K > ma.x{p,, 772} 


When si, S2 G (0, 2) but si 7 ^ S2, we assume further that one of the following holds: 
(b.l) A> 77 , l</3<2 or 

(b. 2 ) A = 77, min{a, /3} < 2 or |min{a, /?} = 2 , K > max{ 77 i, 772, 771 , ^2} / 


/3 = 2 

At > max{77i,771} 


(b.3) A< 77 , l<a<2or< 

{ k > max{772, 772} 

Where the constant 771 is that one given by [40, Lemma 6.5] and 772 is the constant 
given by [40, Corollary 6.4]. 

Then problem (1.14) possesses a nontrivial ground state solution {u, v). 


2 Preliminaries 

2.1 Eigenvalues and compact embedding 

Assume 0 < cr < 2, |0| < 00 , then € L^{il). By [34, Lemma 2.13], the 
functional ^ 

X : dI’‘^{^) R with x{u) = [ T^dx 

Jn FI 

is weakly continuous. Define ( 77 , v)a ■= Jq j^dx, then it is easy to check that (•, •)cr 
is an inner product. We say that u and v are orthogonal if and only if {u, v)^ =0. 

Proposition 2.1. Assume that Ct C K'^(iV > 3),0 < ct < 2 and |n| < 00 , then the 
following eigenvalue problem 

-A 77 = A-^, 17 G (2.1) 
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possesses a sequence of eigenvalues such that 


0 ^ ^ ^^2,(7 fi ‘ ' fi ^k,a ^ ^k+l^a ^ * * * j 

where each eigenvalue is repeated according to its multiplicity. Moreover, oo 

as k ^ oo. Let 62,0-, 63^0-, ■■■ be the corresponding orthonormal eigenfunctions, 
then ei is positive and others are sign-changing. 

Proof. It is well known for the case of cr = 0. When 0 < cr < 2, we refer to [38, 
Corollary 4.2]. □ 

Define E := n if |f^| < 00, we assume that A* > —Ai^^o^ 

then it is easy to see that 

\\u\\E:={lj\Vu\^ + y-l^)dxy ( 2 . 2 ) 

is a norm, and it is equivalent to the norm of 11 It 11 := (iVupda::) ^ due to Proposition2.1. 
Similarly, we define F := fi L^(fl, and if |n| < 00, we assume that 

p* > then 

\\vy-.= [ljVv\^+p*^)dxy (2.3) 

is a norm which is also equivalent to the norm of ||z;||. In present paper, if |0| = 00, 
we always assume that A* > 0, p* > 0. 

Definition 2.1. (see [23, 24] and also[35]) Assume {pk} is a bounded sequence in 
and pk > 0 satisfies 


llPfclUi = A + 0(1), A > 0. 


(2.4) 


Then we call this sequence {pk} is a tight sequence if\/ e > 0,B R > 0 such that 


'\x\>R 


Pk(x)dx < e, V fe > 1. 


We call Uk is a tight sequence, if\uk f is a tight sequence. 


(2.5) 


Remark 2.1. It is well known that the embedding ^ is not compact 

for 2 < p < 2*. However, we have the following compact embedding result for proper 

P- 

Proposition 2.2. Assume a, s G [0, 2). Furthermore, suppose either 

(i) 2 < p < 2*(s) if a < s or 

(ii) 2 + IfEl <P< 2*(s) if a > s. 
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Set E := D, 


1,2 fmN 


) nL^ 


pAf 


dx 


) with the norm 


\m\E ■■= 




Then 


(E, II 


sN 


dx 


( 2 . 6 ) 


is a continuous embedding. Moreover, when a < s with 2 < p < 2* (s) or 0 < s < a 


with 2 


G N-2 


zji < P < 2*(s). the embedding (2.6) is compact. 


Proof. It can be proved by a modification of [38, Lemma 3.1 and Lemma 3.2]. How¬ 
ever, for the convenience of the readers, we prefer to give the details. 

Define 

_2A^+2p—A^p—2s 

•“ 2^ ’ 

_A^p+2s—per —2A^ 

2-g ’ 

_(2A^+2p—Np—2s)cr 

A-2g ’ 

._ 4s+A^pcr —2p<7—2A^cr 

■“ 4-2cr ■ 

Then we have the following results: 


(2.7) 


'0 < Pi < 2,0 < P2,Pl +P2 =P, 

CTl = ^Cr, (71+02= S, 


( 2 . 8 ) 


Then, by the Holder inequality, we have 


[ L£<ix=/' 

i.» kl- Jm 


u 


\P1 


u 


\P2 


-dx 


RAT la;^! |x|'^= 

2 , £L 


< 


\U\ 


RN FT 


-dx 




2-pi 


-dx 


(2.9) 


R« 


By the Hardy-Sobolev inequality, there exists C > 0 independent of u such that 

Pi. p2 

l^kp < C'|u|,^’;2l|w||^, (2.10) 


here ||m|| = 


I Vupdx) ^ and the best constant C satisfies 


When s = (T = 0, it is just the well known Gagliardo-Nirenberg inequality [10, 28]. 
Next, we assume that {u„} C y^) H is a bounded sequence. 

Without loss of generality, we may assume that ^ u in L^(R^, j^) n 
and Un ^ u a.e. in 
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Case 1: a < s,2 < p < 2*(s). Fix tr < s < s, then we see that 2 < p < 2*(s), by the 
embedding result, we obtain that there exists some C > 0 such that 


t-n|s,p < C for all n. 


( 2 . 11 ) 


Then we can see that 


'\x\>R F 


-dx < 


s -- - Jls-S 


l\x\>R Ft 


-dx —>■ 0 as i? —cx). 


( 2 . 12 ) 


Hence is a tight sequence. On the other hand, since p < 2*{s), for any fixed 
i? > 0, we have that 


J\x\<R Ft 

It follows from (2.12) and (2.13) that 


-dx —> 0 as n —>■ c». 


(2.13) 


-dx ^ 0 as n —>■ cxD. 


/BN FI 

Hence, the embedding n Dq'^{R^) ^ Lp(R^, t|^) is compact. 


(2.14) 


Case 2: 0 < s < cr, 2 + < p < 2*(s). In this case, we can fix some s S (0, s) 

such that 

then by the embedding result, there exists some C > 0 such that (2.11) holds. Then 
(2.12) is also valid. Hence, in this case, we also obtain that jllL is a tight sequence. 

\X\ 

Then similar to the previous case, we can also prove the same conclusion. □ 


2.2 Applications on a single equation without critical terms 

Based on the compact embedding result in Proposition 2.2, we obtain the following 
results. 

Theorem 2.1. Let A > 0. Assume that either 

(i) 0 < a < s < 2,2 < p < 2*(s) or 

(ii) Q<s<a <2,2 + ^§^<p< 2*{s), 

then there exists a ground state solution to the following problem 

f-Au + inR^, 

< (2 151 

\u>0inR^,uG Dl'^{R^)nL'^iR^,-^). 

Proof. One can prove it by a modification of [38, Lemma3.4]. □ 
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Remark 2.2. For the equation (2.15), when a = s = 0, it has been studied in [17]. 
The result of Theorem 2.1 above covers the case of a > 0,s > 0. But, the case of 
a > 0, s = 0 /s still open. 

Theorem 2.2. Assume that either 

(1) 0 < O' < s < 2,2 < p < 2*(s) or 

( 2 ) 0<s<(T<2,2+|f^ <p<2*(4 

then the best constant in (2.10) can be achieved. 

Proof. This is a kind of CKN inequality with interpolation term. And the result is 
covered by [38, Theorem 1.1]. □ 

In order to obtain multiple solutions, we now recall the the Krasnoselskii Genus 
defined for studying the even functional 'k on a Banach space E. Set 

A := {A C E\A is closed, A = —A}. 

For A <z A, A f define 



inf{m : 3 e C°(A; M™\{0}),/i(-u) = -h{u)} 
oo, if {••} = 0, in particular, if A 9 0, 


and define 7(0) = 0. 

Theorem A (cf. [31, Theomre 5.7]) Suppose G C^(^M) is an even functional on 
a complete symmetric —manifold M C £’\{0} in some Banach space E and 
suppose 'k satisfies {PS) condition and is bounded from below on M. Let f{M) = 
sup{7(Ar) : K G M compact and symmetric}. Then the functional A! possesses at 
least f{M) < oo pairs of critical points. □ 

As a simple application of Theorem A, the following results are well known. Con¬ 
sider the problem 



(2.16) 


where 12 is a bounded domain in and / satisfies the following assumptions; 

(/i) f{x,-t) = -f{x,t)\ 

(/2) f{x, u) = o{u) uniformly in a; as u 0; 

(/a) / is continuous and \f{x, m)| < a(l + for some a > 0 and 2 < p < 2*, 

where 2* = 2N/{N - 2) for iV > 3 and 2* := -boo for N = 1,2. 

Then for any A > 0, problem (2.16) admits infinitely many distinct pairs of solutions 
(see [31, Themre 5.8] for instance). We note that when 0 G 12 and f{x, u) = “ 

with s > 0, (/a) fails. We also note that the case of |12| = oo will bring some truble. 
However, based on the Proposition 2.2, we can obtain the following result which seems 
to be new. 
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Theorem 2.3. Let C be an open set with 0 S and |0| < oo or |r2| = cxd. 
Furthermore, 

(1) if\Ll\ < oo, we assume thatO < s < 2,2 < p < 2*(s); 

(2) if |r2| = oo, we assume that either 0 < a < s < 2,2 < p < 2*{s) or 0 < s < 
a<2,2 + l^<p<2*is). 

Then equation 

\ —Am + Ar-^ = “ in Ll, 

) ’ (2 17 ) 

\M = 0oMar!, ueD^-^(n)nL^(n,j^). 

possesses infinitely many distinct pairs of solutions for any X > — Aict(A2) if\Ll\ < oo 
and for any A > 0 i/'|f2| = oo. 

Proof OnE = n L^{n, ^), let 

-If (|Vm|2 + x\u\‘^)dx, M:={uGE: ||u|U,p = 1}. 

By Proposition 2.2, we see that the continuous embedding E ^ LP{Q, j^) is com¬ 
pact. Hence, it is easy to see that 'I' satisfies (PS) condition and bounded from below 
on M. By Theorem A, admits infinitely many distinct pairs of critical points on M. 
After scaling, we obtain infinitely many distinct pairs of solutions for (2.17). □ 


3 Single equation involves critical exponent 


3.1 Splitting results for a scalar equation with unbounded domain 

It is well known that, when the nonlinear terms are critical, the corresponding func¬ 
tionals do not satisfy the {PS) condition. Usually, we need a better description of the 
behavior of the [PS) sequences of the corresponding functionals. For the case of a 
scalar equation with a bounded domain, some splitting results have been established 
by Cerami-Zhong-Zou in [7, Theorem 3.1], Ghoussoub-Kang in [11, Theorem 3.1]. In 
this subsection, we are trying to extend this kind of results to the cases of unbounded 
domain in this subsection. We consider the problem 


+ A 1 ^ 1*1 — msi 


|x| 

u{x) = 0 on dVl, 


(3.1) 


where Ll is an open unbounded domain in {N > 3). When 1 < p < 2*(si) — 1, if 
n = or a cone, it is easy to apply the Pohozaev identity to obtain the nonexistence 
of nontrivial solutions (see Corollary 5.3). However, when fl is unbounded but not a 
limit domain(i.e., H is not a cone), there seems to have no related results. Hence, we 
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are aim to extend this work to the unbounded domain case. Firsly, we will prepare 
some results which will be useful. 


When ini = oo and 1 < p < 2*(si) — 1, we can not ensure that / -—;—dx is 

Jn kr" 

well defined for all u S Dg’^(n). Therefore, problem (3.1) have to be posed in the 
framework of the Sobolev space 


E ■= Dl’‘^{n)nLP+^ (^n, 


dx 


(3.2) 


It is easy to see that E is weak close due to the Fatou’s Lemma. We take the norm in E 

11“"" Ua (I 

It is easy to see that when |n| < cxd, E = D^’ (17). When p = 1, it has been defined 
by (2.2), but for the convenience, even p ^ 1, we still prefer to adopt the denotation E. 
To (3.1) there corresponds energy functional, defined in E, is 


<f{u) 




A f |u|P+i 1 f |m|2*('’=) 

Jn ~ 2^ Jn 


(3.4) 


which is of class C^(i5,R). Precisely, we will obtain the following splitting result. 
Compared with the bounded case, they are consistent in the presentation. 

Theorem 3.1. (Splitting theorem for unbounded domain 17) Assume that O C is 
a open unbounded domain with 0 € dfl and dfl is at 0, the mean curvature is 
negative, i.e., E[{0) < 0. Suppose that 0 < Si < 2, 0 < S2 < 2,1 < p < 2*(si) — 1. 
Let {u„} <Z E be a bounded {PS)c sequence of the functional ip, i.e., ip{un) —> c 
and ip'{un) —> 0 strongly in E* as n ^ oo. Then there exists a critical point of 
if, a number k G N, k functions , ■ ■ ■ , and k sequences of radius {r^)n, > 
0,1 < j < k such that, up to a subsequence if necessary, the following properties are 
satisfied. Either 

(a) Un ^ in E or 


(b) the following items are true: 

(bl) G Z7g’^(K(^) C Z7 q’^(K^) are nontrivial solutions of 


2 *(» 2 )- 2 , 





(3.5) 


(b2) ri^ ^ 0 as n ^ oo; 


(b3) 




0, where || • || is the norm in 


14 














m wr^Y!;=Au=\?; 


(b5) ip{u„) <f{U°) + Y!j=iI{U^) with 


I{W) > ( i-^ 

^ ^ 2 2 *[s2 ) 




N ) 2 h 11)-2 ^ 


and 

I{u) := - / \Vu\^dx - for u S 

2Jrn 2*(s2) Jrn |a;|*2 


Remark 3.1. What surprising us is that the representation of the splitting result for 
unbounded domain are exactly the same as that one for bounded domain case (see [7, 
Theorem 3.1]). However, on the applications, the assumptions we required are distinct 
to ensure the “bounded PS sequence” exists. For example, in Theorem , when |11| = 
oo, we always assume that A > 0. 

In order to prove the above theorem, we need to prepare beforehand several lem¬ 
mas. 


Lemma 3.1. (See [7, Lemma 3.2, Remark 3.2] and [11, Lemma 3.3]) Let Ll be an open 
subset ofS.^. 

1) Assume that {un} C LP(Ll, j^), 1 < p < oo. If{un} is bounded in L^ (Ll, -0^) 
and Un ^ u almost everywhere on LI, then 


-dx — 


In W 


-dx 


-dx t 


In Fl' 


2) Assume {m„} C Dq’ (LI) is such that Un ^ u in Df (LI), then 


/ |Vu„pda; — / |Vm„ — Vu\'^dx —>■ / \Vu\'^dx 

»/ Q Q J ^ 


as n ^ oo. 


d) If Un —^ u weakly in Dg’^(R^''), then 


P "^Un |Un — Up ^(Un—u) |up 


as n ^ OO 


in H 


-1 


□ 

Lemma 3.2. (cf. [7, Lemma 2.1]) Let Ll C be a set having Lebesgue measure 
|f2| < oo. Let 0<s<2, l<p< 2*(s). Then the embedding Hq(LI) ^ L^(Ll, 0^) 
is compact. 
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Lemma 3 . 3 . Consider {«„} C E such that I{un) —>■ c and I'(un) —^ 0 in E*. 
For r„ G (0, cxd ) with r„ —>■ 0, we assume that the rescaled sequence Vn{x) := 

Tn ^ Un{rnx) is such that Vn ^ V weakly in Dq’ (R.^) and Vn ^ v a.e. on R^. 
Then, I'{v) = 0 and the sequence 

2-iV j; 

Wn{x) := Un{x) - rn^ v{ -) 

Xn 

satisfies I[wn) c — I{v), I'{wn) 0 in E* and ||ui„|p = ||uri|P — ||f ||^ + o(l). 

Proof. We note that when is a bounded domain, this lemma is given in [7, Lemma 
3.3], a similar result we refer to [11, Lemma 3.4]. Following the procedure carefully, 
we see that the proof of [7, Lemma 3.3] is still valid when fl is unbounded. Hence, 
we omit the details. We shall give the detailed proof about this type of result for the 
system case, see the forthcoming Lemma 4.8. □ 

Indeed, compared with the bounded case, the biggest difficulty is that how to ensure 
^ 0 which is a necessary condition to apply Lemma 3.3 to continue the splitting 
iteration. When |H| = oo, let be a bounded {PS)c sequence for /.p, and Un ^ u 
in E, it is not trivial that — tt) —?► 0 in £^*. To overcome these difficulties, we 
require the following results: 

Lemma 3 . 4 . Consider the equation (3.1). Assume that 0 < si < 2,0 < S 2 < 2,p < 
2* (si) — 1, then —|~|S 2 — j ^ tight sequence if{un} G E is a bounded sequence. 

Proof. Since p + 1 < 2*(si), we have 


2N - 2si - N{p + 1) + 2{p + 1) > 0. 


Since S 2 > 0, we can take cr € (0, S 2 ) such that 


Define 


_4-2si_ 

!Ar-2si-Ar(p+l)+2(p+l) ’ 

2si(s2-ct) _ r, 

2si-tV(p+l)+2(p+l) ^ 

2(s2-<t) _ r. 


„ 2 (p+l)(s 2 -a-) 

■ 2Af-2si-Ar(p+l)+2(p+l) ’ 

P2 := 2*(s 2) - (pi + 1 ) - 1 , 

_ ._ _ 2si{s2-<7) _ 

!• 2Af-2si-Af(p+l)+2(p+l) ’ 

02'.= a — (Tl, 

- (p+l)<T2 

P-Pl 


(3.6) 


(3.7) 
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then we have 


0 < Pi + 1 < min{p + 1 , 2*(s2) - 1 }, 
0 <P2 + l,Pi+P2= 2*(s2) - 2, 

<7i > 0, CT2 > 0, CTi + CT2 = O', 

0 < CT < 2, 

(p+i)(P2+i) _ 2*(a) 
p-pi ' '' 

Then by the Holder inequality, for any A C H, we have 


(3.8) 


|2*(s2) 


/A FI' 


-dx = 


< 


/a 

IP +1 ) 


dx 


u‘ 


Pl+1 

P + 1 


IA \x\^^ P1 + '- 

f |m|p+i 


—dx 


\u\ 'p-Pl 


P—PI 
P+1 


dx 




-dx 


Pl + 1 
P+1 


/ ]+_ 

'a k 


2* (a) 


-dx 


P —PI 
P+1 


Hence, by the Hardy-Sobolev inequality, we see that |rtra|CT.2*(s2) bounded due to the 
boundedness of {u„} in E. Hence, 


|2*('<2) 


-dx = 


tnn FI' 


/ ^ 

./b“ no 


|2*(^<2) 


-dx 


<- 


FI' 

|2*('i2) 




-da; 


■nn FI' 


which implies that 


\Un 


|2*(s2) 


—^0 as R —^ cX), 


is a tight sequence. 


□ 


Lemma 3.5. Consider the equation (3.1). Assume that 0 < si < 2,0 < S2 < 
2,p < 2*(si) — 1 and that {it„} C E is a bounded {PS)c sequence for ip{u), then 




KF+M . 


7 


is a tight sequence. 


Proof. Let xr{^) ^ be a cutoff function such that 0 < XR F 1. Xr{^) = 0 

in Bp{, xr{^) = 1 in and |Vxrj| < Then it is easy to see that UnXR € E- 
When {un} is a {PS)c sequence, we have 

lUnK'^XR 


/ VUn ■ V{UnXR) + ^ / 
Jq, Jq. 




a FI' 


= o(1)||u„||b. (3.9) 


Since {un} is bounded, going to a subsequence if necessary, we may assume that 
Un ^ u in E, Un ^ u a.e in —i> u in for 7 £ [1,2*). Then for any 

fixed i? > 0, we see that 

lim [ UnVun-VxR= [ uVu-Vxr- (3.10) 
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By the absolute continuity of the integral, we have 


lim / uVm • Vx_R = 0, 

R-^oc Jq 


(3.11) 


because of that 


uVu ■ Vxr 


<2 

<C 


L 


R<\x\<2R 


R 


|Vu| < 8 


Ir<\x\<2R Ft 


IR<\x\<2R 


|Vu|^ 


'R<\x\<2R 


\Vu\^dx. 


By (3.9), we have 


^ XR = o(l)l|wn||j 


|2*(s2) 


In FI' 


-XR- / UnVUn-VxR- 


Recalling that {u„} is bounded in E, by Lemma 3.4, we obtain that 
tight sequence, which implies that 


| 2 *(* 2 ) 


(3.12) 


lim 


|2*('<2) 


R^ooJq |x|®2 

By (3.10)-(3.13), we have 


lim limsup / ( |Vu„p + A 

_R—).oo n—).oo Jq \ 


Xr = 0 uniformly for all n. 


F ' 


Xfl = 0 


and it following that \ -1 i 

' fI®! j 


is a tight sequence. 


(3.13) 

(3.14) 

□ 


Lemma 3.6. Assume that 0 < si < 2,0 < S2 < 2,1 < p < 2*(si) — 1, and that 
{un} C E is a bounded [PS)c sequence for ip{u) such that Un ^ u in E, then up to 
a subsequence. 


\Un\^ ^Un |w|^ 


in E* as n —t oo. 

Fl“i FT'^ 

Proof. For any h G E and e > 0, by Lemma 3.5 and Holder inequality, there exists 
some > 0 such that 


[ 

^\Un\^ ^Un - |w|^ ^U\ 

JnnM% 

tie 

1 1 1 rLCtcc 

V j 


< 


< 


/nnB'i 


\Un\^ ^Un, 


, f 

|m|^ 

dx + 


JnnM% 

die 

a; 




dx 


/nnB= FI 

He 


/nnB?, FI 


E 


<e\\h\\E. 


(3.15) 
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On the other hand, by Lemma 3.2 and Holder inequality, we have 


\Un\^ ^Un - \u\P 


/nnBRg 

By (3.15) and (3.16), we prove that 


hdx 


= o(1)||/i|1b as n ^ cxD. (3.16) 


\Un\P ^Un | m |^ 




in E* as n ^ oo. 


□ 

Corollary 3.1. Assume that 0 < si < 2,0 < S2 < 2,1 < p < 2*(si) — 1. Let 
{un} <Z E be a bounded {PS)c sequence for ip{u) such that Un ^ u in E, then 


In 


dx—^ —;—dx as n ^ oo 


and 


Ur^^u in LP+1 . 


Proof. Rewrite 

f \Un\P^^ — \u\P~^^ f {\Un\P~^Un — \u\P~^u) Un f — u) 


Since {«„} C i5 is a bounded {PS)c sequence for (p{u), by Lemma 3.6, we have 

r f 


In \x[ 


dx —^ —;—dx as n —>■ OO. 

In 


Combined with Lemma 3.1, we obtain that 


—>■ M in PP'^^ ( n 


dx 


□ 


Proof of Theorem 3.1. The proof is very like to [7, The proof of Theorem 3.1]. 
However, since H is unbounded, there will be some differences in the details. Hence, 
for the convenience of the readers, we will give the details. 

Let {un} C E he a bounded {PS)c sequence of <p(u), that is, <f{u) —>■ c and 
p'{un) —?► 0 in E*. Recalling that E C is weak close, then there exists 

G Hq{LI) so that, up to a subsequence, Un ^ in E and Un a.e. on R^. 

For any h G E, by Lemma 3.6 we have 


-dx 


\U°\P-^U°h 


dx as n oo. 
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Combined with Lemma 3.1, we see that {(p'{U^),h) = 0 since ip'{un) —0 in E*. 
Thus, = 0. Let := Un — U^, if |fl| < cxd, by Lemma 3.1 and Lemma 3.2, it 

is easy to see that the sequence u\ := Un — satisfies 


\<\\ = \\u\\\e + 0(1) = \\uJe - ||C/°|b + 0(1) = ||U„|| - ||C/°|| + 0(1), 
/'(u)j) 0 in 

J{ui) Cl :=c-ip{U°), 

(3.17) 

1 

where ||m|| := ( \S/u\'^dx) ^. If |n| = oo, by Lemma 3.6, Corollary 3.1 and Lemma 
3.1, we can also obtain the results of (3.17). 

If —>■ 0 in ZlQ’^(fl), by Corollary 3.1, we obtain that in E and we are 


done. If 74 0 in ^^’^(n), we claim that 


lim inf 

n—>-oo 


U. 


1 |2*(s2) 


-dx > 0. 


'n FI' 
, 1 . 2 / 


(3.18) 


Otherwise, the facts that C E C Dq (O) is bounded and/'( u)j) —OinTf ^(O) 
would bring to a contradiction because 

{I'{ul),ul)= / \Vui\^dx- / ' dx^o, 

Jn Jn FI 

and, hence, > 0 in Oq’^(O). Thereby, (3.18) holds true. 

Define an analogue of Levy’s concentration function 


Qnir) := [ 


dx. 


Fr 


Since Q„(0) = 0 and, due to (3.18), Qn{oo) > 0 , we can choose 


< — lim inf 

2 n—foo 


\U, 


1|2*(s2) 


-dx 


so small that 0 < <5i < ^ and find a positive sequence {r^^} such that 

Qn(?'n) = ^1- Setz;i(a;) := (r^)^u),(rix). Since Hu^H = |K|| is bounded, we 
may assume in w), a.e. on and 


<5, = 


,1|2*(s2) 


-dx. 


(3.19) 


We claim that ^ 0. 

Set and let /„ G Ilg’^(O) be such that {I'{u\)^ h) = f Vfn ■ Vh for 

Jn 

any L € L>o’^(0). Then p„(x) := (r)j)'^/„(7’ia;) satisfies f = [ |V/„p 

7n„ Jn 

and (/'(u^),/i) = / Vp„ • V(i for any/i G i9Q’^(r2„). 

Jn,, 
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If =0 would be true, then we could deduce that, for any h G with 

supp /i C Bi, 


[ |V(hui)p=/ Vu^V(h\l)+o(l) 

J IB IBi 

r P*(S2) f 


<Ms,(R^) 


N\-l 


V. 


2*(s,)-2 

1|2*(s2)\ 2-(S2) 


' S2 


/supp h 1^ 

2-32 

r>N\-lcN-S2 / |V7/';...1m2 


[ |V(hui)|%o(l) 

«/IBi 


=PsMn-'sr^^ / iv(K)r+o(i) 

«/IBi 

<11 |V(hui)|2 + o(l), 

" »/ IBi 

here we are using the inequality: 


h^\ 


2*(s) 


< 


Is.^ fI 


N\-l 


|2*(s) \ 2*(«) 


’supp h fI 




|v(MI^ 


that holds for all M G and h G C“(R^) (see [11, Lemma 3.5.]). 

Hence, —)• 0 in (Mi) and, by the Hardy-Sobolev inequality, —>■ 0 

in ^Bi, which follows a contradiction with (3.19). Thus, the claim is 

proved. 

Now, let us prove that 0. We proceed by contradiction. Otherwise, by the 

choice of (5i and Lemma 3.4, we obtain that {r^} is bounded. Hence, we may assume 
that rij —>■ > 0, then the fact that ^ 0 in would imply u^(x) := 

u)^(r}^x) ^ 0 in Oq’^(R'^), in contradiction with ^ 0. Therefore, —>■ 

0 . 

Next, we prove that supp C R+ . Without loss of generality, we assume that 
dR^ = {xn = 0} is tangent to dfl at 0, and that —bn = (0, • • • , —1) is the outward 
normal to dfl at that point. For any compact K C RiY, we have for n large enough, 
that ^ n if = 0 as —>• 0. Since supp C ^ and F- a.e. in R^, {7^=0 
a.e. on K follows, and, therefore, supp C R+ . 

By (3.17) and Lemma 3.3, we obtain that = 0. Hence, is a weak 

solution of (1.3). Moreover, by Lemma 3.3 again, we see that the sequence u^{x) := 
u)^(x) — also satisfies 

(\Kr = \\uj^-\\uY-\\uT + o{i), 

ll{ul)^C2:=c-piU°)-IiU^), (3.20) 

[l'{ul)^0mH-\n). 


21 









We also note that 


HU^) > ( 2 - 


2*(s2) 


^^S2( 


i>N\ 


2* (s2) 

’{ s 2)-2 


By iterating the above procedure, we construct critical points of I(u) and sequences 
(r^) with the above properties. Here we note that since dx = o(l), ||u^|| = 

ll'^^ralls + o(l)’ we see that {u^} is also can be viewed as a bounded {PS)ci for <f{u) 
with Cl = c — and Ci = c — — I(U^) — ■ ■ ■ — i > 2. Hence, 

by Lemma 3.4, we can guarantee the boundedness of {r^}. And since (p{un) c, the 
iteration must terminate after a finite number of steps. □ 


3.2 Existence results on a single equation 

3.2.1 A supplement to a single equation on the domain with finite Lehesgue mea¬ 
sure 


Firstly we will study the case of p = 1 basing on the result of Proposition 2.1 as a 
supplement to the results of [7](see [7, Remark 2.1]). That is, we consider the following 
problem 


'-Au + A*^=aH 
u{x) = 0 




in n 
on dfl, 


(3.21) 


After rescaling properly, we may assume that A = 1. We obtain Theorem 1.1 and 
Theorem 1.2. 

Proof of Theorem 1.1: When H is a bounded domain, it is just a special case of [7, 
Theorem 1.4]. We only need to prove that u > 0. Since the process is very like, we pre¬ 
fer to give the detail of this part in the proof of Theorem 1.2. When H is unbounded, we 
only need to replace [7, Theorem 3.1] by Theorem 3.1 above, by the similar arguments 
as the bounded case, we can obtain the results. □ 

Proof of Theorem 1.2: Without loss of generality, we may assume that A = 1. Define 


(p{u) 



2*(si) 


dx, 


which is of class C^(i7Q(H),R). Since A* > —Ai_sj(0), by Proposition 2.1, we can 
choose the norm 

which is equivalent to the usual one in i7g (H). Recalling that 2*(s2) > 2, it is easy to 
verify that (p has the mountain pass geometry. Now, we define the mountain pass value 


c* := inf max 

7erie[o.i]^^ 


(3.22) 
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where T := {j{t) G C'([0,1], iJg (fi)) : 7(0) = 0, (^(7(1)) < 0}. Then there exists a 
(PS')c* sequence, that is, a sequence {un}m Un G such that 


$(u„) -)• c*, 


By the Hardy-Sobolev inequality, we see that c* > 0. On the other hand, following the 
processes of [7, Lemma 2.4.], we can prove that there exists some vg G iTg(O)\{0} 
such that (p(vo) < 0 and 


max ip(tvo) < I - 


1 

2^ 



Hence, we have 


0 < c* < 


2*(si) 


Msi( 


,, 2*(si) 


It is easy to check that {«„} is bounded in Hq{H). Then by [7, Theorem 3.1] if H is 
bounded and by Theorem 3.1 if H is unbounded, there exists some G TTg (O) such 

that Un -> t/° in i7g(fl) and ip'{U^) = 0,(p{U'^) = c*. Hence, is a solution of 

(3.21). Now, let 

£/ := {u^ 0,ip'iu) = 0 }, 


then 


^ 0. Define cg = inf (p{u), then by Hardy-Sobolev inequality, we see that 


DiV^ 


2*(gl) 


by [7, 


Cg > 0. Let {u„} C .e/ be a minimizing sequence, then {m„} is a bounded {PS)c 
sequence of (f. Note that 0<co<c*< (- - —^ 

\2 2*(si) 

Theorem 3.1] or Theorem 3.1 again, up to a subsequence, uq, a ground state 

solution of (3.21). Without loss of generality, we may assume that itg > 0. Now, we 

leta(.):=-r^ + M"'-’-= 

a weak solution of 


-, it is easy to check that a{x) G and itg is 

—Am = a{x)u,u G i7g(H). 


Then the Brezis-Kato theorem [4] implies that Mg G for all 1 < r < 00 . Then 

Mg G for all 1 < r < 00 . By the elliptic regularity theory. Mg G 

Finally, by the maximum principle, we obtain that Mg is positive. □ 


3.2.2 The cases ofp = 1 with |H| = 00 orp > 1 with unbounded domain ft 


In the following, we are 
lowing problem 


concerned with the existence of positive solutions to the fol- 


—Am + A 




u{x) > 0 in H, 
u{x) = 0 on dil, 


U^*G2)-1 

|xp2 


in H, 


(3.23) 


where H is an open unbounded domain in N > 3,2*(s2) := ■ When 

p < 2*(si) — 1, there seems to have no results when fl is an open unbounded domain 
butH Here we are aim to give something new. We obtain Theorem 1.3. 
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f 

When 1171 = oo and 1 < » < 2 *(si) — 1 , we can not ensure that / —;—dx is 

well defined for all u G Dq (fl). Hence, Problem (3.23) is posed in the framework of 
the Sobolev space 


E;=D«(n)nJ7+' (s!.i^). 


It is easy to see that E is weakly closed due to the Patou’s Lemma. We take the norm 

\SIu^dx 


\u\\e ■■= 


|y|p+i \ p+i 

-dx 


In 


In this case, we always assume that A > 0. 


When |H| < oo, we always assume thatp > 1 since p = 1 has been 

studied in subsection 3.2.1. To (3.23) there corresponds the variational functional, 
defined in E, by 

which is of class C^iE, R). 

Lemma 3.7. Assume that il C R.'^ is an open unbounded domain and 

X > 0,1 < p < min{ 2 *(s 2 ) — 1, 2*(si) — 1} if |H| = oo; 


either A < 0,1 < p < 2*(si) — 1 or 

^ A > 0,1 < p < min{2*(s2) — 1, 2*(si) — 1} ;/ |H| < oo. 

Then for any u G LiyiO}, there exists a unique tu > 0 such that 
tuU G JV := {u f 0, J{u) = 0}, 

where 


J(u) ■.= {lp'{u),u) = / \Vu\^dx-\-X 


-dx — 


| 2 *(^< 2 ) 


-dx. 


In Fl' 


in Jn FI ' 

Moreover, Af is close and bounded away from 0. 

Proof The proof of existence and uniqueness of is standard, we omit the details 
and refer to [39, Lemma 2.1]. And it is trivial that Af is closed. The proof of that Af 
is bounded away from 0 is a little different, hence we prefer to give the details of this 
part. 

For any u G Af, we have J{u) = 0. Then by the Hardy-Sobolev inequality, for the 
case of A > 0, we have 

[ \X/u\'^dx< [ iVupdx + A [ 

Jn Jn Jn Jn 




N\-l 


\Vu\'^dx 


2 *(« 2 ) 
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and for the case of A < 0, |ri| < oo,p > 1, by Lemma 3.2, we also have 


iVupda; = f 

Jn 


u 
n 


2*(s2) r |u|P+i 

dx — X -—;—dx 


In 


2 *(s 2 ) 


<(Hs,(M ) / +C|A| / \Vu\^dx 


Hence, there exists some i5o > 0 such that ( / \\/u\‘^dx) ^ > Sq and it follows that 


Hub > iVul^dxJ >So>0. (3.24) 

Hence, Af is bounded away from 0. □ 

Lemma 3.8. Assume that H C is an open unbounded domain and 


X > 0,1 < p < min{2*(s2) — 1, 2*(si) — 1} if |H| = cxd; 


either A < 0,1 < p < 2*(si) — 1 or 
A > 0,1 < p < min{2*(s2) — 1,2*(si) — 1} if |H| < cxd 


Then any {PS)c sequence of p is bounded in E and any {u„} C Af, a {PS)c sequence 
is also a {PS)c sequence of p. 

Proof We firstly consider the case of |H| = cxd and A > 0,1 < p < min{2*(s2) — 

l, 2 *(si)-l}. 

Le {un] C i? be a {PS)c sequence of p{u), that is 


Denote a„ 
we have 

and 

Hence, 



\ p'iun) 0 in E*, 

I p{Un) C. 


|VuJbx,6„:= f ^dx,cr.:= ( 


{.P iPrifUn) — Un “f Xbn Cn — o(l)||Un||i? 

, , 1 A, 1 

= 7:“" + -On - TTTt - \^n = C+ o(l). 

2 q 2 *(s 2 ) 

2 ^) ■ 2 ^) ^ ■ 


(3.25) 

(3.26) 


(3.27) 
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1 1 

Noting that ||m„||£; = ai + bn, without loss of generality, going to a subsequence if 
necessary, we may assume that 


if i 1 

an > -^WUnWE or > -\\Un\\E- 


If an > ^||uri||_E, recalling that A > 0, g < 2*(s2), we have 


1 


2 2*is2)J 4 


1 , 


-7\\Un\\E< T;- 


1 1 


2 2*(s2) 




2 2"(»2) 

=C + o(l) (1 + llWnlU) , 


q 2 *{s 2 ) 


Xbn 


and if 6^ > i ||u„||£;, we have 


1 1 


q 2 *{s 2 ) 


\Un\\E 




q 2 *is 2 ) 


<|'i- ^ 


Xbn 


an + 


.2 2*(s2) 

=C + o(l)(l+ \\Un\\E), 


1 1 


q 2 *{s 2 ) 


Xbn 


which implies that {tt„} is bounded in E. 


Secondly, we consider the case of |fl| < oo. We note that if A > 0, g < 2*(s2) , 
the arguments displayed above are valid. Next, we assume that A < 0, 2 < g < 2*(si). 
Notice that in this case, the embedding iAg’^(n) ^ compact and the 

norm ||u„||£; is equivalent to the usual one ||w„|| = an - Hence, note that 


2 2 

o(1)||u„||b + 2c + o(l) =-Xbn - - rC„ - Xbn + C„ 


2*{s2) 

= ( 1-^ |A|6„ + ( 1 ~ 


2*(s2) 


we have 


llunlll; ^CUn — C (o(l)||u„||£; + C„ + |A|h„) 
<C (^o(l)||u„||£; + (1 - ^)|A|6„ + (1 - 
<C (o(l)||u„||_E + 1) , 


which also implies that {it„} is bounded in E. 

The remaining proof is similar to [39, Lemma 2.3], we omit the details. □ 
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Lemma 3.9. Let C be a open domain with 0 € dfl. The boundary dfl is 
at 0 with the mean curvature H (0) < 0. Assume that 

rA>0,l<p< min{ 2 *(s 2 ) — 1,2*(si) — 1} if |ri| = cxd; 


either A < 0,1 < p < 2*(si) — 1 or 
X > 0,1 < p < min{2*(s2) — 1,2*(si) — 1} if |i2| < oo. 


Furthermore, if X > 0, we require that p < 


N-2si 

N-2 


enough. Then we have 0 < Cq < 0 



or p > with |A| small 

where 


Co := inf 09 (m). 
u£N 


(3.28) 


Proof Under the assumptions, by Lemma 3.7, the corresponding Nehari manifold is 
well dehned, hence cq is well dehned. By (3.24), we obtain that cq > 0. When O is 
bounded, cq < 0 is given by [7, Lemma 2.4]. And we note that the proof is completely 
valid for the unbounded case, since the construction of testing function only required 
the local information of U near x = 0. □ 


Proof of Theorem 1.3:Let {it„} C Af he a minimizing sequence, then 


I <p{un) Co := inf„g^(/j(u) 

^ip'^Un) —?> 0 in E*. 

By Lemma 3.9, we have cq € (0, 0) and by Lemma 3.8, {un} is bounded in E. 
Then by Theorem 3.1, up to a subsequence, there exists some u G E such that u 

strongly in E and (p{u) = cq. Hence, u G Af isa minimizer. Without loss of generality, 
we may assume that u > 0 is a ground state solution of (3.23). Finally, it is standard to 
show uo is positive. □ 


3.2.3 Li-Lin’s Open problem on unbounded domain 

Li-Lin’s Open Problem (see [19, Remark 1.2]) For the situation si < S 2 and A < 0, 
the existence of positive solutions to 


Au + X 


,2*(n)-l 


„ 2 *(<. 2)-1 


= 0 


Ixpl ^ \x\‘2 

u{x) > 0 in LI, u{x) = 0 on dLl. 


in Ll, 


is completely open. Even for the equation 

u2*(s)-l 

Am — vF -\ -;—;-= 0 in LI, (3.29) 

III'* 

where 0 < s < 2 and 2*(s) — 1 < p < the existence problem still remains an 

interesting open question. □ 
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The first partial answer to Li-Lin’s open problem is obtained in [7], where the 
authors consider the following problem: 


Am + + 




u{x) > 0 in n, 
u{x) = 0 on i90, 


= 0 


in n, 


(3.30) 


where is an open bounded domain in N > 3, 2*(s2) := ■ When 

2*( 52 ) — 1 < P A 2*(si) — 1 , it is a case considered in the above mentioned Li- 
Lin’s open problem. For such cases, we see that the functional still possesses the 
mountain pass geometric structure. We also note that the method of Nehari manifold 
fails now and one can not ensure any {PS)c sequence is bounded any more. Hence, the 
Ekeland’s principle is not enough for this problem, to overcome this difficulty, one thus 
needs to develop more sophisticated critical point theories which assert the existence of 
a bounded {PS) sequence. Cerami-Zhong-Zou obtain the following results thanks to 
the perturbation method and the well-known Struwe’s monotonicity trick (see [15, 30]). 


Theorem A1 (cf. [7, Theorem 1.5]) Let H C be a bounded domain such that 
0 € dfl. Assume that dfl is of at 0 and that the mean curvature ofdLl at 0, H{0), 
is negative. If 

A<0, 2*(s 2) - 1 <p < 2*(si) - 1, 

then there exists Aq < 0 such that (3.30) has a positive solution for all Aq < A < 0. □ 


Theorem A2 (cf. [7, Theorem 1.6]) Let H C be a bounded domain such that 
0 £ dfl. Assume that dfl is of at 0 and the mean curvature of dLl at 0, 7T(0), is 
negative. If 

A<0;2*(s2)-1<p<^^^, 

then for almost every A < 0, (3.30) has a positive solution. □ 

Next, we are going to study the Li-Lin’s open problems on the unbounded domain 
Q. The framework of the Sobolev space E = Hq{LI) if |r2| < 00 and E = DQ’^{n) n 
LP+i(H,p^)if|H| = 00 . Usually, the nontrivial weak limit m of a {PS) sequence 
Un is a solution we are searching for. The perturbation method state out the weak limit 
is nontrivial through the geometrical position without the splitting result. 

Theorem 3.2. Let U C be a unbounded domain with |U| <00 such that 
0 £ dfl. Suppose that dLl is of at 0 and the mean curvature of dLl at 0, H{0), is 
negative. Let assume 


A<0, 2*(s 2)-1 <P<2*(si)-l, 

then there exists Aq < 0 such that (3.30) has a positive solution for all Aq < A < 0. 

Proof. Replace Hq{LI) by E and the arguments of [7, Theorem 1.5] are completely 
valid. We omit the details. □ 
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Recalling Lemma 3.9 is still valid for unbounded domain. We see that following 
results obtain in [7] hold. 

Lemma A3 (cf.[7, Lemma 6.1]) Consider the problem (3.30) with 2*(s2) — 1 < p < 
■ Then V A < 0, there exists 0 < b < |A| such that for almost every p G 
[A — i5, A + 5], there exists a bounded sequence {u„} C i7o (fl) ({rtn} C E) with 

I^iun) ^ < 0 =: (i - 

Cn > 0 and /'(un) 
h{u) := 

□ 


^0ini7-i(n) (/;(u„) ^ 0 in L;*) Where 


1 f iVupdx-^ 

2 7n p + 1 


,,p+i 


1 


2*(^<2) 


In khi 2*{s 2) Jn 


-dx 


Corollary A4(cf.[7, Corollary 6.1]) For almost every A < 0, there exists a bounded 
sequence {«„} C iTo(fl) ({un} C i?) such that Ix{un) c\ < Q and —>■ 

0mH-\n) {r^{un) ^ 0 in E*). □ 

Recall that Theorem 3.1 gives a splitting result for the case of unbounded domain, 
we can also obtain the following theorem: 

Theorem 3.3. Let Lt C be a unbounded domain such that 0 G dVl. Suppose 
that dfl is of at 0 and the mean curvature ofdLl at 0, 77(0), is negative. Assume 
further 

A<0,2*(s2)-1<p<^^^, 
then for almost every A < 0, (3.30) has a positive solution. 

Proof. By Lemma A3 and Corollary A4, for almost every A < 0, there exists a 
bounded (PS')c^ sequence with 0 < Ca < 0. Then, up to a subsequence, there exists 
some U\ G E such that ^ U\. We only need to prove that U\ 0. Indeed, assume 
U\ = 0, since p < 2*(si) — 1 we can apply Theorem 3.1 obtaining fc > 1 and 

fc k 

CA = /a(u„) + o(l) = /a(La) + ^ /([/") + o(l) = ^ /([/'=) + o(l) > 0, 

i=l i=l 

a contradiction. Hence, a positive solution U\ to problem (3.30) is found. □ 

Remark 3.2. Compare Theorem 3.2, Theorem 3.3 with the ones for bounded domain 
case, i.e.. Theorem A1 and Theorem A2. What surprising us is that the representation 
are exactly the same even for the range of p. Now, let us give an explanation. When we 
apply the perturbation argument or monotonicity trick, the framework of work space 
is E and thus the bounded [PS)c sequence obtained are relate to the norm || ■ lb- 
Another related explanation see Remark 3.1. 
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4 Compactness results for the system case 


4.1 The case oia + j3 < 2*(s2) and |fi| < (X) 

Lemma 4.1. Suppose that C with the Lebesgue measure |ri| < c» and 0 < 
S 2 < 2,1 < a, 1 < /3,a + /3 < 2*(s2). Let {(■«„, Vn)} C ^ := x be a 

bounded sequence such that (m„, Vn) (u, v) in then 


ItinT ‘^Un\Vn\^ 

X p's ^ 

|u “ ‘^u\v\^ 
|x|*2 

in H ^(H) 

(4.1) 

\uff\Vn\^~'^Vn 

M “|v 

in 

(4.2) 

X ^*2 

1x1*2 


Proof. We only prove (4.1). Since |n| < oo, there exists some i? > 0 such that 
|n n B^l < 1 and f - —j— dx < 1 for S 2 > 0. Note that S L^{Lt Ci B/j) 

since fl fl B/j is bounded domain in and S 2 < N. Hence, -j—j— € L^(0). By the 
absolute continuity of the integral, we obtain that 


/Ano 


1 


dx ^ 0 as meas{K) 0. 


(4.3) 


Without loss of generality, we may assume that u,Vn x in H up to 

a subsequence. Thanks to the Egoroff Theorem, for any e > 0, there exists some 
Hi C H such that |Hi | < e and 


—)■ u, Un —>■ u uniformly in H\Hi as n —>■ oo. 
ForV h G HqIQ), we have 


|Mn|“ '^Un\Vnf 




hdx 


<f 

lUnT ^Un\Vnf ^ 

dx + 


jQi 

X *2 

Jqi 

X *2 


dx 


IWnT '^Un\Vnf 


|u|“ 


/n\ni V Ft 
:=/ + // + III. 


hdx 


(4.4) 


Recalling that {un,Vn) is bounded, by (4.3), the Holder inequality and the Hardy- 
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Sobolev inequality, we have 


I = 


' f2i 


|Un|“ ‘^Un\Vn\^ . 


dx 


i 


|a;|®2 

1^1 \Vn\^ 


I I •. , , _^£2 (q!-1)s2 (2* {s2) — ot-li)s2 

Oi |a;| |a;| 


dx 


< 


\h\ 


2*(s2) \ 2*(s2) 




r2i 


|2*(s2) \ 2*(s2) 




|2*(s2) \ 2*(4) 


f2i 


2*is2)-c.-l3 

1 \ 2* (S2) 


=o(l)||/i|| as |fli| —>• 0. 

Similarly, we also have II = o(l)||ft.|| as |fli| —0. On the other hand, for any fixed 
fli, by (4.4) and the Hardy-Sobolev inequality, it is easy to see that 


III = o(l)||/i|| as n oo. 


(4.5) 


Hence, we obtain 


|Mn|“ 1^1“ '^u\v\^ 


which means that 


a; ^*2 

1 

1 

|a :|*2 j 

\Un\°'~‘^Un 

\Vn\^ 

|u U ^ 


hdx 


= 0(1)11/^11, 


in iJ ^(O). 


□ 


4.2 The case of a + /? < 2*(s2) and |f2| = oo 

For the case of |0| = oo, we always assume that > 0, then ||u||£;, ||r'||F defined 

by (2.2) and (2.3) are norms. We denote 


{A%^) either (To < S 2 or 


0 < S 2 < (To, 

2 + + ^ 


/A* • . [ 0 < S 2 < ?7o, 

Lemma 4.2. Suppose that O C with |0| = oo and (^, 70 )’ !<«,!< 

/3, a + /3 < 2* ( 52 )- Let {(u„, Wn)} C ^ : E x F be a bounded sequence such that 
(un,Vn) (u,v) in , then 


m. 


x|“ '^Un\Vn\^ l^r 

I I .1? n I ™ I .« 




(4.6) 
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and 




|y|«|l,|/3 2^ 


in H ^(51). 


(4.7) 


Proof. We only prove (4.6). Without loss of generality, we assume that —?► u 

a.e. in fl. Under the assumptions (A*^), by Proposition 2.2, we see that 

' are tight sequences. Then for any /i € by the Holder inequal- 

ity, we have 




|U„|“ ^Un\Vn\'^ 


|u|“ 


hdx 


<! 

\Un\°‘ ‘^Un\Vn\^ ^ 

dx-\- 

It “ ^u\v\^ ^ 


|x|®2 


X 


dx 


< 


\u^ 


\a+P 


tnn Ft 


-dx 


|a+/3 


a-l 


a-1 

3TF 


Wn 


\a+P 


-dx 


\h\ 


a+P 


-dx 


Bjjnn fI 


Bjjno Ft 


tnn 


a;p2 


-dx 


tnn 


v\o,+p 

|a;p2 




-da; 


|/r| 


a-|-;S 


ST? 


i,nn 


a;p2 


-dx 


= o(l)||/i|| as i? ^ cx). 


(4.8) 


Apply the similar arguments as that in Lemma 4.1, we can obtain that 

Un\°‘~'^Un\Vn\^ |u|““2y|^|/3 


'{kl<fl}nn 


By (4.8) and (4.9), we get (4.6). 


hdx 


= o(l)||(i|| as n oo. (4.9) 

□ 


4.3 The case of ct +/5 = 2 *(s2) 

Lemma 4.3. Let a > 1, then for any e > 0, there exists C{e) >0 such that 


a; + yr {x + y)-\x 


o'-2„ 


< elx 


cr—1 


C{e)\y\ 


cr—1 


(4.10) 


Proof It is trivial for x = 0 or cr = 1. Next, we always assume that cr > 1 and X 0. 
For the case of x > 0, let t = -, then (4.10) is equivalent to the following inequality 


l + f|'^ (l + f)-l <e + C(e)|f| 


CT—1 


(4.11) 


We define 


f{t) ,= \i+tr^{i+t)-i 


t e 


then it is easy to see that f{t) is continuous on R. Notice that /(O) = 0, we obtain that 
for any e > 0, there exists some fo > 0 such that 


/(f) < e if|f|<to. 


(4.12) 
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On the other hand, since 


limsup 

|i 1^00 


fit) 

|i|-i 


= 1 . 


Thus, by the continuity, there exists some C'(e) > 0 such that 


fit) < C'(e)|t|'^ \ |t| > to. 


(4.13) 


Then, (4.11) follows from (4.12) and (4.13). 

For the case of a; < 0, we let f = —4, then (4.10) is equivalent to the following 
inequality 


If-ll"^ (f - 1) + 1 <e + C(e)|f| 


a—1 


(4.14) 


Let 


git) ■= 


\t-ir^it-i) + i, 


f e R, 


the similar arguments also lead to (4.14). 


□ 


Corollary 4.1. Assume that 0 < s < 2,(T > 1. Let {«„} C be a 

bounded sequence such that Un ^ u a.e in then 


\Unf '^Un - Wn - ^ 


{Un - u) - |u 


<7-2 


\x\ 


—^ 0 in L I 


pAf\ 


(4.15) 


Proof. For any fixed e > 0, define 

:= {^\Un\''~‘^Un - \Un - - u) - - e|u„ - , 

then by Lemma 4.3, we have 


ff,<il + C{e))\ur-\ (4.16) 

Notice that 0 a.e in then by Lebesgue’s dominated convergence theorem, 

^g_^0inL^(R^). (4.17) 

1^1 2 * f 1 


By (4.16) again, we have 


|tt„r Un-\Un-u\'^ ^{Un-U)- 


\u\^-^u 


W-1)b 

a; 


< - ’LL - 

- (<T-1)S 

a; 


+ e- 


\Un - u 


a—1 


(<7-l)s 

a; pw 


(4.18) 
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Hence, 


lim sup 


< lim sup 


MnT ^Un-\Un-u\'^ ^ {Un - u) - \u\'^ 


a; =•(=) 


2*(«) 



J n 




n^oo JrW \ I I 2*(s) 


2*(a) 

< 2 


2*(«) 



+ £- 


./ n 


u„ - u 


2*(g) 
cr-l\ ‘'-1 


X 


2*(«) 


2*(s) 
|CT-l\ ■'■^1 


lim sup , . - 

ri->00 JR-N Y Ij;! 2*(<i) 


+ e- 


I I 

|x| 


2*(s) 

|cr-l\ '^“1 


= 2 lim sup / I £l 

n—J-oo 

2*(s) 

< Ce—. 

Now, let £ —i> 0, we obtain (4.15). 


((r-l)a 

U 


□ 


Lemma 4.4. Assume that 0 < S 2 < 2,1 < a, 1 < /3,a + /3 = 2* ( 32 )- Let 

dx . 

be a bounded sequence 


{{Un,Vn)} C L2*(«2) X L2*(^2) 

a; in then we h 
|Mn|“ - \Un - m|“ “ \u 


and u„ u, Vr, t; in K^, f/ien we /zav'e 


q;S2 

\x\ ^*('=2) 


Oin L^-^(R^) 


and 


\v.. 


if - \Vn - V\^ - \v\^ ^ "•(f 2 ) 


—> 0 in L ^ 




(4.19) 


(4.20) 


|a;|5^T^ 

Proof. For any fixed £ > 0, we define 


:= (ll'UnP - \Un - uf - 1141“! - e\Un - < (l + c(£)) K 


and 


5n := - \Vn-vf - - e\Vn - vf'^ < (l + c(£)) |x |^, 

where we use the facts of a > 1, /3 > 1 and 

\x + y|* - |x|‘ < £|x|‘ + c(£)|?/|‘,0 <t < 00 . 

Then by Lebesgue’s dominated convergence theorem, 

J n 


\x\ ^*"°“ 2 ) 


0 in (R^) and -^ 0 in (R^) 


9s2 

|x| =*<“2) 
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Since 

and 

we obtain that 


Unr - \Un - Ur - \ur 


< f^ + e\un - ur 

if - \vn-vf - |t;|^| < g%^ + e\vn - vf, 


limsup / 

n—>-oo 


< limsup 

n—>-oo ^ 


Unf - \Un - uf - \uf 


as2 

\x\ 


2 (s2) 


fn+s\Un - ' 


|a;|2W 


2*(«2) 


< 2 “ 


< Ce. 


limsup / 

n^oc J^N 


2 *(s2 ) 


2*(-i2) 


|a;| 


+ e" 


2*(«2) ktn — U 


- 7;,P*(s2) 


S2 


(4.21) 


[ 

\vuf - 

\Vn - vf - \vf 




Ik’S 


a; =*('’ 2 ) 


2*(52) 


Similarly, 

limsup/ (4.22) 

n->oo jRiV |j.| 2 * (, 2 ) 

where C is independent of n. Let e ^ 0, we obtain (4.19) and (4.20). □ 


Lemma 4.5. Assume that 0 < S 2 < 2,1 < a,l < /3,a + /3 = 2*(s2). Let 

-2*(s2)('Tn)tV 

LI" 


{{un,Vn)} C X ^ bounded sequence and 

a.e. a.e. 

Ur), U. Vy] 


V in then 


''^Un\Vnf |Un-M|“ {Un - u)\Vn - vf |u|“ 


^ 0 in 

(4.23) 


and 


\Un\°‘\Vnf '^Vn |u„ - m|“ |v„ - ^(Un-u) lltl^lul^ “V 




0 in H-^IR^) 
(4.24) 


Proof. We only prove (4.23). Let h £ Ilg’^(R^), it is sufficient to prove that 


C pUnf 

lun-uf '^{Un - u)\Vn - vf 

|u “ 

Jr'S \ a; 

\x\^^ 

) 


h = o(l)||/i||. 
(4.25) 


35 




































We note that for any subset of 


MnT ^Un\Vn 


1^ - \Un - U|“ ^(Un - u)\Vn - v\^ - 


|m|“ '^u\v\^'jh 


[b 


’ 71 1 I ‘^n 


’■{Un -u) - |u|“ 


+ 


f \Un - u\°‘~‘^{Un - u) [\Vnf - \Vn - vf - h 

Jq 

f \u\°'~^u[\vn\^ - \vn - v\>^ - |t;|^] h 

^ Jn 

f \Un - u\°‘~‘^{Un - U)\vfh 

Jn 

r \u\°'~‘^u\Vn - V\^h 

Jn l^h 

:= i{n) + ii{n) + iii{n) + iv{n) + v{n). 

Take O = R^, since Vn is bounded in ( R'^, , | and J^7^\ + n + 

\ ’ U S 2 J 2 *(S 2 ) 2 *(S 2 ) 

= Ij by the Holder inequality, the Hardy-Sobolev inequality and Corollary 4.1, 


we can have 

cATx f [\Un\°‘~'^Un-\Un-u\°'~'^{Un-u)-\u\'’‘~'^u]\Vn\^h 


') = 


X 


S2 


= oil)\\h\\. 
(4.26) 


Similarly, by (4.20), we can obtain that 

II{R^) = o(l)||/i||,//J(R'^) = o{l)\\h\\. (4.27) 

Hence, we only need to prove that 

/C(R^) = o{l)\\h\\ and C(R^) = o(l)||/i||. (4.28) 

By the Holder inequality and the Hardy-Sobolev inequality again, we have 
\Un - u\°‘~‘^{Un - U)\v\^h 




< 


_y|2*(«2)\ / r |/i| 2*(«2)\ / r |^|2 *(s2)\5^ 


/b^ FI' 


/B^ fI 


<C\\h\\ 


|2 *(s2 )\ 


(4.29) 
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Similarly, we can prove that 


Hence, by the absolute continuity of the integral, we have 

IV{M%) = o(l)||/i||, = o(l)||/r|| as i? ^ cx). 


(4.30) 


(4.31) 


For some fixed R large enough, apply the Egoroff Theorem in the bounded domain 
Bij, for any (5 > 0, there exists Hi C B/j such that |H| < S and Un ^ u,Vn ^ v 
uniformly in B/{\Hi. The similar arguments above shows that 

/E(Hi) = o(l)||/r||,E(Hi) = o(l)||/r|| as |Hi| ^ 0. (4.32) 

On the other hand, the uniform convergence leads to that 

/E(B/{\Hi) = o(l)||/i||, E(B_r\Hi) = o(l)||/i|| as rn- oo. (4.33) 
Then (4.28) follows from (4.31)-(4.33). □ 

Lemma 4.6. Assume that 0 < S 2 < 2,1 < a, 1 < /3, a + /3 = 2* ( 52 )- Let 
{{un,Vn)} be a bounded sequence of -j—x -j— 

such that {un,Vn) {u,v) in R^. Then 


lim 

n—)-oo 


/ ( 

^ \Un\°‘\Vn\^ 

\Un - u\°‘\Vn - 

/rJV ' 




dx = 


\u\ \v 


1^ 


dx. (4.34) 


* / dx \ 

Proof. Since is bounded in ( R'^, — ), by 

V Frv 


the Holder inequality and 


(4.19), we have 


[|Un|“ - \Un - U|“ - |ti|“] \Vn\^ 


Similarly, since by (4.20), we obtain that 

f \Un - [lunl^ - \Vn - v\^ - \vf] 


= 0 ( 1 ) as n ^ 00 . 


= 0 ( 1 ) as n ^ 00 


and 


|wr[kn|'^ - \Vn -v\^ - lul^] 


= 0 ( 1 ) asn ^ Qo. 


(4.35) 


(4.36) 


(4.37) 


Fr" 

By the Holder inequality again and the absolute continuity of the integral, we have 


/ 


\Un - m |“| u |^ 


= 0(1), / 




= 0 ( 1 ) as i? —>• 00 . 


(4.38) 
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Apply the Egoroff Theorem in the bounded domain for any (5 > 0, there exists 
some Oi C Bfl such that |f2i| < 5 and —)■ ^ u uniformly in Bfl;\r2i. The 

similar arguments above shows that 


\Un - U V 




= o(l), 


— v\^ 


= o(l) as |fli| —i> oo. (4.39) 


The uniformly convergence leads to that 


Un - Ur\V 




= o(l)> 

It follows from (4.38)-(4.40) that 


\u\ \v. 


— v\^ 


'Mr\Qi 


= o(l) as n —oo. (4.40) 


— v\^ 


= o(l) as n —> oo. (4.41) 


/R« \xr Js.^ Fr^ 

Then by (4.35)-(4.37) and (4.41), we obtain (4.34). The proof is complete. 


□ 


Corollary 4.2. Assume that 0<S2<2, l<a, l</3, « + /? = 2*(s2). Let 
{{un,Vn)} C ^ bounded sequence 

such that {un, Vn) (u, v) in R^. Furthermore, if Un u or Vn v strongly in 

j^2*{b2) 


dx 


, then 


Iwnr '^Un\Vn\^ 


|u|“ . 

. . in li. 


-imiVx 


(4.42) 


and 




M U 


\P-\ 


in H 


pAf\ 


Proof. They are straightforward conclusions from Lemma 4.5. 


(4.43) 

□ 


Corollary 4.3. Suppose that Lt C R'^ is an open set and 0 < S 2 < 2, 1 < a, 1 < 

P,a+l3 < 2*(s2)- Let {{un,Vn)} C ^ be a bounded sequence such that {un,Vn) ■ > 

(m,u) in n. Furthermore, for the case of a + 13 < 2*( 32 ), we assume that either 

|n| < 00 or 1121 = 00 with (^170 )• L'tirthermore,for the case of a + /? = 2* ( 52 ). 

dx 
u i 


we require that m„ 
u = 0, we have 

and 


or Vn V strongly in L^ ^12, ■ 


|Mn|“ '^Un\Vn\^ 




0 in H-\n) 


Oin i7"Ff2). 


Then if u = 0 or 

(4.44) 

(4.45) 


Proof It is a direct conclusion from Lemma 4.1, Lemma 4.2 or Corollary 4.2. 


□ 
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4.4 Splitting results 

Next, we will establish the splitting theorem for the system case. Assume that 17 C 
is an open domain with 0 € 917 such that 917 is at 0 and the mean curvature of 917 
at 0 is negative. 0<S2<2,l<a, l</3)Q! + /3< 2* ( 52 ) and one of the following 
holds: 


(i) 17 is bounded, 0 7 ^ A* > — Ai_cro(A7), 0 7 ^ p* > — Ai^^o(17); 

(m) 17 is unbounded but |17| < 00 , 0 7 ^ A* > —Ai,cto(^)iO 7 ^ /r* > —Ai_^u(17); 

{iii) |17| = 00 , A* > 0, p* > 0 and conditions (A*^) and (A*^) are satished. 

Remark 4.1. Since the domain we considered can be bounded or unbounded, and for 
the unbounded case, the measure is also allowed to be finite or infinite. The conditions 
we required will change a little in different situations. For example, when |17| < c», we 
consider X* > — Ai^o-o (^); (17). While |17| = 00 , we only consider X* > 

0,/i* > 0, moreover, we require the assumptions (A*^) and (A*^) if a + jd < 2 *(s 2 )- 
For the convenience, we prefer to denote our work space by the same denotation Q) := 
E X F. When |17| < 00 , it is easy to see that E = F = TJq (17). 

We denote the least energy corresponding to (1.7) by Cao,\*,\, then 


-1^ _ 1 
Ccro,A*,A —[ry 


2*(si) 


) / \S/u\‘^ +X*Pf-dx 


, -2 /I 1 

=A=*('’i)-2- 

^2 2*(.i) 

=A^"^c,„,am. 


/ \Vuf + X*-^dx 

In 


Hence, Cao,\*,x is continuous and strictly decreasing relate to A S (0,+oo). By [7, 
Theorem 4.1], any ground state solution of (1.7) is a mountain pass solution. 

Remark 4.2. When X* > 0, the assumption 77(0) <0 can guarantee 


2*(si) 


Psi (A7) 


< c^o.am < 1 2 2*(si) 




\ 2 —Si 


And when |17| < 00 , our assumption is A* > —Ai^o-j, (17), i.e.. A* is allowed to be neg¬ 
ative. When A* < 0, it is easy to prove that Ccto,a*,i < 


The details of this kind of estimation we refer to [7]. 
For any u G E,v G F, we dehne 

|2 


2*(si) 




'f'A(tr) :=i^(^|Vup + A* 


\xA° 


dx — 


X 


|2*(si) 


2*(si) in l®h 


-dx 




.|2*(si) 


2*(si) 


-dx. 
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For any (it, v) G 


$(ii, v) := 4'a(m) + T^(i;) - k f dx. 

Jn \x\^ 


(4.46) 


Lemma 4.7. Suppose that C is an open domain with 0 G Oil such that Oil 
is at 0 and the mean cun’ature of dfl at 0 is negative. 0 < S 2 < 2,1 < a, 1 < 
/3, a + /? < 2* ( 52 ) and one of the following holds: 

( 1 ) il is bounded, 0 7 ^ A* > — Ai^o-o(^2), 0 f p* > — Ai^t^q (il); 

(2) n is unbounded but |r2| < cxd, 0 ^ A* > — Ai^o-q (^2), 0 f p* > —Ai,,,(, (f2); 

(3) |r2| = 00 , A* > 0, p* > 0, especially when a + /3 < 2* ( 52 ) we require further¬ 
more and {A*J. 

Let {{un,Vn)} G ^ be a bounded {PS)m sequence such that {un,Vn) (u,v) 
in Furthermore, if a -\- P = 2* ( 32 ), we assume further that Un ^ u or Vn ^ v 
. / dx \ 

strongly in f f2, -j—)• Then going to a subsequence if necessary, 

define 

rhi := lim 'I'A(Mn) 


we can 


and 


m 2 ■■= lim T^(i;„). 


Moreover, if u = 0 or v = 0, then the following are satisfied: 

(i) m = rhi + m 2 - 

[ii) {itn} is a {PS)mi sequence for ^'a and {wn} is a {PS)m 2 sequence for T^. 


{Hi) either rhi = 0 or mi > Cao,\*,\; either ■012 = 0 or m2 > 

Proof Since {(itn,Vn)} is bounded in it is easy to see that \E'A(un) is bounded, 
so does Tfj_{vn). Hence, up to a subsequence, rhi and m 2 are well defined. Since 
^'{un, Vn) 0 in we have 


.T,/ t.. ^ 

^X\'u^n) — - 1 — -—^ 0 ^ 


X 


S2 


and 


- kP 


\Un\°‘\Vn\^ "^Vn 


Oin F*. 


Hence, if it = 0 or 1 ; = 0, by Corollary 4.3, we have 


r ,T,/ t 1 r „ 

lim lim -^^-= 0 

n—fco n—¥oo 
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and 


lim T' (vn) = k/ 3 lim 


/- 


= 0 . 


Hence, (ii) is proved. We also have 

\Un\°'\Vn\^ 


j 

JQ. 


dx = o(l)||r;„|| —>■ 0 


In 

due to the boundedness of {u„}. Then it follows that 

m= lim ^{un,Vn) 


= lim T'a(m„) + lim T„(u„) - k lim / 

n—¥oo n—^oo n—^oo Jq 




dx 


n m' 


=rhi + 7712- 


By [ii), it is easy to see mi > 0, 7 ti 2 > 0 since 2*(si) > 2, A > 0, ^ > 0. If mi ^ 0, 
then Un 74 0. By the definition of Cao,\*,x, we see that 


xh\ ^ 


Similarly, if m 2 7 ^ 0, we have 

^2 ^ ,fJ.- 


□ 


Corollary 4.4. Under the assumptions of Lemma 4.7, let {[un,Vn)} be a bounded 
[PS)m sequence with 


0 < m < min{c^o,A*.A,c^o,A«*,A«} ■ 


Then $ possesses a nontrivial critical point [u, v),i.e., u f 0,v f 0. 

Proof. It is an easy conclusion directly from Lemma 4.7. □ 

Next, we define 


h[u) = ^ / iVupdx - —^ / 

2 2 (si) Jr 


|2*(si) 


-dx 


Fl' 


(4.47) 


and denote the corresponding ground state value by mx. Then, a direct calculation 
leads to that 

'I 1 


mi = 


and 


2 2*(si) 

mx = A2’(oi)-2mi. 




Corollary 4.5. Under the assumptions of Lemma 4.7, let {(u„, )} C St be a bounded 
sequence such that ^ [u,v)- Furthermore, we assume that Un u or 

Vn ^ V Strongly in 14^^) If ol + /? = 2*(s2)- We have the following results: 
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(i) ifu = 0, then either rhi = 0 or rhi > m\, 

(ii) if V = 0, then either m 2 = 0 or m 2 > rn^, 
where rhi , ^2 (ire defined in Lemma 4. 7. 

Proof Recalling that {zi„} is a {PS)mi sequence for '1 'a and Un ^ u in E = Hq{LI). 
By [7, Theorem 3.1 ] or Theorem 3.1, we obtain that either —?► it in i? or there exists 
k functions U^, - ■ ■ ,U^ such that 

k 

= ^\{u) + ^ Ix{U^) + 0 ( 1 ), 
i=i 

where u is a critical point of T'a and Ufj = 1,..., fc are critical points of Ix. Hence, 
we have 

^x{Un) = ^' a ( u ) + 0 ( 1 ) 
or 

'^x{Un)>'^x{u)+Ix{U^)+0{\). 

Especially, when u = 0, we have 

'^xiUn) = 0 ( 1 ) 


or 

Hence, 


^aK) > Ja(C/') + o(1). 


rhi = 0 or mi > mx- 


Similarly, we can prove that if u = 0, we have 


m 2 = 0 or m 2 > m^. 


□ 


Remark 4.3. By Remark 4.2, we see that Corollary 4.5 is an improvement of Lemma 
4.7. And naturally we have the following result as an improvement of Corollary 4.4. 

Corollary 4.6. Under the assumptions of Lemma 4.7, let {{umVn)} be a bounded 
{PS)m sequence for $ and 

0 < m < min{mA, mil}. (4.48) 

Then $ possesses a nontrivial critical point {u, v),i.e., u 0,v 0. 

Proof. It is trivial by Corollary 4.5. □ 

Remark 4.4. Noting that for the critical couple case, i.e., a + P = 2*(s2), we as¬ 
sume in Lemma 4.7 that ^ u or —>■ u strongly in if ;—V This 

\ krv 

assumption is very strong, hence the Corollary 4.6 above is not enough for the case of 
a -\- P = 2*(s2) in applications. We need a better description of the behavior of the 
(PS) sequences of the functional 4>. 
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Here comes our main splitting results. We will discuss it by two cases of a + /3 < 
2* ( 52 ) and a + /? = 2* ( 52 )- The idea of proof is similar to the single equation case, 
however it is much more complicated for the system case. Hence, we will give the 
details. 

Theorem 4.1. (Splitting for a + (3 < 2* {S 2 )) Assume that O C is an open domain 
with 0 G 9H, dfl is at 0, 0 < S 2 < 2,1 < a, 1 < /3, a + /3 < 2*(s2) and one of the 
following holds: 

(1) H is bounded, 0 7 ^ A* > — Ai_cro(^)!0 p* > 

(2) fl is unbounded but |H| < 00 , 0 7 ^ A* > — Ai_cro(H),0 p* > —Ai_,,(, (fl); 

(3) |r2| = 00 , A* > 0, p* >0 and conditions ) and (A*^) are satisfied. 

Let {{un, u„)} be a bounded {PS)m sequence. Then there exists a critical point (u, v) 
of <i> and two numbers k,l € N U {0} and functions U^, ■ ■ ■ , • ,V^ and 

sequences of radius 1 < j < fc, 1 < i < I such that the following properties 

are satisfied going to a subsequence if necessary: 

(a) either {u,v) in ^ or 


(bl) Vn ^ V in F and S C iAg’^(K^) are nontrivial solutions of 




|x| 

= 0 on dM.^. 


(4.49) 


(b2) —>■ 0 n —>■ c»; 


(b3) 




0, where || • || is the norm in 


(b4) + 


(b5) ^{un,Vn) - $(u,u) h{U^) or 


(cl) Un ^ u in E and 14* g C Dg’^(R^) are nontrivial solutions of 


|„| 2 *(. l )-2 


+ " =0 hiR^, 


\ V = 0 on 


(4.50) 


+ • 


(c2) —>• 0 as n —>• 00 ; 
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(c3) 


— V — 2 V'^{—) 0, where W ■ W is the norm in 


(C4) IknlP^IMI^+EUll^iP/ 


(c5) $(m„, Vn) - $(m, v) J2i=i or 


(dl) are nontrivial solutions of (4.49) and are nontrivial solutions of (4.50); 


(d2) —>• 0, /ojj —>■ 0 as n ^ oo; 


(d3) 


Uu-u-Y.U^ri)^W[^) 

0, where || • || is the norm in 


0 and 


Vn-V-Y.i^liPn) " 


(d4) wur^r ^ hip+ Y.U ^ ii^ii'+sLi 

(d5) Vn) - Hu, v) ^ h{U^) + ELi HiVH 

Proof. Since {{un, t’n)} is bounded in up to a subsequence, there exists some {u, v) 
such that (un,Vn) {u,v) in S>. Since {(u„,u„)} is a {PS)m sequence of $, it is 
easy to see that v) = 0. Then one of the following holds: 

(a) {un,Vn) {u,v) in 


(b) Un H u in E but Vn ^ v in F. 


(c) Un ^ u in E but Vn yf u in F. 


(d) Un yf u in E and u in F. 

We only prove the case of (&1) — (65) which is corresponding to the case of (b) above. 
Since u„) —>-0, we have 


1- ,T,/ ^ ^ l^"l“ '^Un\Vn\^ ^ 

lim ^xHn) - na -j—n-= 0. 

n—foo 


Use Lemma 4.1 if |U| < c» and apply Lemma 4.2 if |U| = oo, we have 


lim 

n—¥oc) 


Un\Vn 




S2 


|m|“ 


(4.51) 


(4.52) 
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Noting that <&'(«, v) = 0, we have 




|m|“ ^u\v\^ 


(4.53) 


Combined with (4.51), (4.52) and (4.53), we have 

lim = 4 'a(m)- 


(4.54) 


By 3) of Lemma 3.1, we have 




a; ri 


a; ®i 


->■ 


a; 


Hence, 


lim 4'A (un — u) 

n—>-oo 

= lim ( — A(u„ — u) + A 
n—>cx) V 


,, {Un -u) _ ^\Ur,. - U 


— 7/.P — 


{Un - U) 


a; 


= lim 

n—¥oo 


M ^ ^ * (Wn - m) , , 

-A(u„-u) + A —f-T^--A( 




|2*(si)-2 


p.) 


= lim 4 'a(w«) - 4'a(m) 

n—¥oo 

= 0 (by (4.54)). 


Let Wn = Un — u and c = m — $(«, v), then {wn} is a {PS)c sequence of 'I'a. Notice 
that Wn 0 but Wn 7 ^ 0 in E. Hence, apply [7, Theorem 3.1] if H is bounded and use 
Theorem 3.1 if H is unbounded, we obtain that the case of (61) — (65) happens. 

By the similar arguments, if u„ u in but 74 u in F, we obtain (cl)-(c5); if 
Un jA u in E and 74 u in F, (dl)-(d5) hold. □ 


Next, we are going to establish the similar splitting result for the system case with 
critical couple terms. Denote 

A{u, v) := Ix{u) + I^{v) - K [ dx. 

Remark 4.5. Noting that & := E x F, next we will also adopt the notations := 
Dl-'^(R^)xDl’\R^)and\\{(l>,ij)f := [ {\Vcl)\'^ + dx while := 

Jn 

/ (|V(/)|^ + |V'0|^) dx. For the convenience, we prefer to use the common nota- 

JR-N 

tions A(u,v) and A'{u,v) for different domains. However, we will distinguish them 
by the words. For example, if for any {h,h) g we have {A'(u,v), (6,6)) = 0, 
then we will say that A'{u,v) = (0,0) in if for any {h,K) g we have 
{A' {u, v), (6, 6)) = 0,then we will say that A'{u, v) = (0, 0) in 4^*. And if we only 
say that A'{u,v) = (0,0), that means for any {h,H) g Zlg’^(R(^) x Dq’'^{R^), there 
holds {A'{u,v), (6,6)) = 0. 
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Lemma 4.8. Assume that O C is an open domain with 0 G did, dil is at 0, 

0<S2<2,l<a, l</3, a + /3 = 2* ( 52 ) ond one of the following holds: 

( 1 ) il is bounded, 0 7 ^ A* > —Ai_cro(^)j0 p* > —Xi,rio(id); 

(2) il is unbounded but |r2| < 00 , 0 7 ^ A* > — Ai,cto(^)i0 p* > —Ai_^(,(f2); 

(3) |rj| = 00 , A* > Q,p* > 0. 

Let {(un,Vn)} C ^ such that A{un,Vn) c and A'(un,Vn) —>■ 0 in For 
{?■„} C (0, + 00 ) with r„ —>■ 0, assume that the rescaled sequences 

N-2 N-2 

4>n{x) := rn^ Un{rnX), tpn{x) ■■= Tn^ Vn{rnX) 


are such that 


and 

{4’n, f’n) if, 'f) on 

Then A'if, tp) = 0 and the sequences 

2 — N ^ 2 — N ^ 

Xn(x) := Unix) - T~ fi -), (finix) := Vnix) - r~ V’( —) 

Xn Xn 

satisfy 

Aixn, Tn) c- Aiu, v), A'ixn, Tn) ^ 0 in 

and 

IKXnj ^■n)\\gs = IKWra, 'yn)||^ ~ +o(l), 

where 

\\if,n\%--= i i\vf? + \vf\^)dx. 

JRW 

Proof Without loss of generality, we assume that dR^ = {xn = 0} is tangent to 
at 0, and that —cn = (0, • • • , —1) is the outward normal to dil at that point. It 
is standard to prove that supp f, supp ip C (see [11, Lemma 3.4] or [7, Lemma 
3.3]). Hence, we may assume that (0„, ipn) if, 'f) in Z?g’^(R^) x £)g’^(R^). It is 

easy to see that Aiu, v) is invariant under dilation and we have 

N-2 

\\iXn,Pn)\\% =\\Xn^ {xuiXnX), PniXnX))\\%, 

= \\ifn - f,-fu - f)\\% 

= \\ifn,'fn)\\% - \\if,'f)\\%+0il) 

= \\iUn,Vn)\\% — II (</>,'*/') II + o(l). 

By Lemma 4.6 and Lemma 3.1, we have 

Aifn - f,'fn - 'f) = Aifn, 'fn) “ Aif, f) + o(l). 
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Hence 


N-2 N-2 

=A((/>„ -(j),1pn- V') 

=A{4>n,'4ln) - A{4>, Ip) + o(l) 

=A{Un, Vn) - A{(p, Ip) + o(l) 

=C- A{(p, 1 p) +o{l). 

Forany/i,/le CpP^{R^),M hn{x) := (r„)^/i(^),/i„ := then we 

have that hn x hn € ^ for n large enough due to the assumption that r„ 0. Thus 

{A'{cp, Ip), {h, h)) ={A'{(pn,tpn), {h, h)) + o(l) 

= (-A Vn), {hfi, ^n)) “t” 

=o(l)||(/i„,?i„)|p + o(l) 

=o{l)\\{h,h)\\%, + o{l), 


which implies that A'{(p, ip) = 0. 

N-2 

Next, for any {h, H) G let hn{x)) := Xn ^ (/i(r„x), fi(r„x)), then we 

can see that for n large enough, supp /i„, supp hn C 1R+. By Lemma 4.5 and Lemma 
3.1, we have 

A'{(pn,1pn) - A'{(pn - (p,ipn - ip) - A'{(p,ip) ^ 0 in (4.55) 

Hence, for {h,h) G we have 
{A'{Xn,‘Pn), ih,h)) 

N-2 N-2 

= {A'{rn^ Xn{rnX),rn^ Pn{rnX)){hn,fln)) 

= {A'{cpn - Cp,1pn- Ip), (hn, K)) 

= {A'{(pn,1pn), {hn,K)) - {A'{(p,1p), (L„,ft„)) 

+ o[l)\\{hn,K)\\'%, (by (4.55)) 

= {A\(pn,ipn),{hn,K)) + o(l) || (/i„ ,/!„)||^ {sincs A'{(p, Ip) = 0) 

= {A'{Un,Vn),{h,h)) + o{l)\\{hn,K)\\% 

= o{l)\\{h,h)\\% since = \\{h,h)\\%. 


□ 


Now, it is enough for us to prove the splitting result for critical couple case. 

Theorem 4.2. (Splitting Theorem for the critical case a + /3 = 2*(s2)) Assume that 
n C is an open domain with 0 G dVl, dTl is af 0, 0 < S 2 < 2,1 < a, 1 < 
/3, a + /3 = 2 *(s2) o.nd one of the following holds: 

(1) Q. is bounded, 0 ^ A* > — Ai.o-o(H), 0 f p* > — Ai^^p(H); 
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(2) is unbounded but |r2| < cxd, 0 A* > — Ai^o-q (Ai), 0 ^ > —Ai_,,(, (f2); 

(3) |rj| = oo. A* > 0,/^* > 0. 


Let {{umVn)} C 3! be a bounded {P S)c sequence for the functional i.e., <&(«„,w„) —^ 

c and ^'(un,Vn) —> 0 strongly in 3* as n ^ oo. Then there exists a critical point 
V^) of a number fc G N, fc pairs of functions (J7^, V^)] • ■ • ; (C/^, V^) and 
k sequences of radius {r-^)n','i"n > 0; 1 ^ ^ ^ such that, up to a subsequence if 

necessary, the following properties are satisfied. Either 

(a) {un,Vn) ^ {U'^,V°) in 3 or 

(b) all the following items are true: 

(bl) (0,0) 7^ (UfV^) e -Do’^(R^) X c X 

are critical points of A; 

(b2) ^ 0 as n ^ oo; 

0, where || • \\^ is defined in Lemma 4.8; 


(b4) \\{u^,vn)\\% ^ E-=o \\{u^.v^)\\%; 

(b5) Vn) ^ $(C/°, FO) + Ejti A{w,v^) with 

A{U\V^)>c^>0, 

where Coo := mf{A(M, v) : {u, v) (0,0), A'[u, v) = 0}. 

Proof. Let {(««, Vn)} C ^ be a bounded {PS)c sequence of $(m, v), that is, 

f$(u„,u„) -^cinM, 

0 in 3*. 

Up to a subsequence, there exists some (f/°, V^) € 3 such that 

f (tt„, Vn) (t/°, V°) in 3, 

\{un,Vn) ^ (C/0, UO) on 


Then it is easy to see that $'(C/0, f/O) = 0. Denote Un '.= Un — C/O, := — V^, 

then by Lemma 4.5 and Lemma 3.1, we have 

f U<,vi)\\%, = \\{Un,Vn)\\% - \\{U,V)\\%+0{1), 

lA'iulvi)^0in3*, (4.57) 

[Al«,t;i)^c-<i>(C/0,U0). 
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If (u^, v^} —>■ Oin we are done. If (u^, v^) /)■ 0 in we claim that 


lim inf 

n —>‘00 


,l|2*(si) 


n 




-dx + 


/ -da;) > 0. (4.58) 

in ^ 


,1 „i 

^rf> 

would bring to a contradiction: 


Otherwise, the facts that u^)} C ^ is bounded and that —>• 0 in 




= [ (|V<|2 + |Vz;i|2)da;-(A / ^^^dx + f, 
an ^ an 




-dx 


In FI' 


- K(a 13) f 
an 




n F 


S2 


^ 0 . 

On the other hand, by [40, Corollary 2.1 or Corollary 2.2], we have 


lim inf 


In F 


S2 


-dx = 0 


and 

lim inf / -da; = 0 

in 

and then it follows from the Holder inequality, we have 


\K{a + /3) 


,1 |a| 




da; I 0. 


Hence, 0 in Thereby, (4.58) holds true. Define an analogue of Levy’s 

concentration function 


Qnir) 


|a;|'*i 


|a;|®i 


dx. 


Since (5n(0) = 0 and, due to (4.58), Qn{oo) > 0 , we can choose di > 0 small enough 
such that 




and find a positive sequence {r^} such that Qn 


= di. Set 


{Un{x),vl,{x)) := ((r^)"='ui(r^a;),(ri)“ 



Since II 11^ = || (u^, ||^ is bounded, we may assume 

in dI’'^(R^) X «, vi) ^ {U\V^) in and 


di 




dx. 


{U\V^) 

(4.59) 
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Since 0 < si, S 2 < 2, by [40, Corollary 2.1 or Corollary 2.2] again, we can also choose 
(5i small enough such that 


2*(..) ^5- 

2 -(«) 


(4.60) 


(4.61) 


We claim that ((7^, V^) ^ (0, 0). 

Setl2„ = ;^flandlet/„ G be such that w^), (/i, 0)) = [ V/„ • 

" Jn 

V/i for any/i G Do’^(fl). Then g„(x) := fn{rnx) satisfies / |Vg„|^ = 

Jq.„ 

/ |V/„p and (fi, 0)) = [ Vg„ • V/i for any fi G £>o’^(n„). 

JO ./ o.^. 

If =0 would be true, then we could deduce that, for any h G 67“ (K^) with 
supp h C Ml, 


[ \y{hul)\^= [ Vu^V(fi^ui)+o(l) 

«/IBi «/]Bi 




l|2*(2i2) 


< 


|a;p 


+ / V 5 „ • \/{h‘^ul) + Ka 


+ 0(1) 


l«l.',lJ/3/,2 


/ Bi 


■o(l) 


, I i„ a f II P f 

+ \K\a -—,—- / --h \K\a- 


< 


2*(s2) 

N\-l 


2 *{s2 ) 




• h Ft 


|V(M),)|^ + o(l) 


N\-l 


2*{s2) 

\K\aP 






supp h l^l 


■ 


l^ihulW 


N\-l 






,h FI 


-) 


1 m2 


2*(s2) 

<jf +o(l), 

^ JBi 

here we use the Young inequality and the following inequality; 


|V(Mi)| 


Jr^ |x| Jsupp h 1^1 Jr^ 


2*(s) 2 *(s )-2 


which holds for all u G I7o’^(R^) and h G 67“(see [11, Lemma 3.5.]). 
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Hence, Vu^ 0 in ( 


L). Thus, by the Hardy-Sobolev inequality. 


i). Similarly, if = 0, we have —?> 0 in 

2*(si) /'^ 


0,vl —>• 0 in T 


loc 


51, ■ 


which follows a contradiction with (4.59). Thus, the claim is proved. Now, let us prove 
that 0. Otherwise, when O is bounded, we see that {r^} is a bounded sequence. 
And if O is unbounded, under our assumptions, similar to Theorem 3.1, we can still 
guarantee the boundedness of {r^} when 6i is small enough. Up to a subsequence, we 
may assume that —>■ > 0, then the fact that (u^, v^) (0,0) in ^ would imply 


(un(x),Vn(x)) ■■= (0,0) in O, 

in contradiction with (U^, V^) ^ (0, 0). Therefore, —>■ 0. 


o’^( 




1>N\ 


Next, we prove that supp C , supp C . Without loss of generality, 
we assume that dR^ = {xat = 0} is tangent to 90 at 0, and that -Bn = (0, • • • , -1) 
is the outward normal to 90 at that point. For any compact K C R(Y, we have for n 
large enough, that fl iT = 0 as r)j —>■ 0. Since supp C ^, supp u), C ^ and 
(u^.vl.) (U^, V^) a.e. in R^, (U^, V^) = (0,0) a.e. on K follows, and, therefore, 

supp C/i C R);!, swpp C R(^. 


By (4.57) and Lemma 4.8, we obtain that A'{U^,V^) = 0. Hence, {U^,V^) 
is a critical point of A. Moreover, by Lemma 4.8 again, we see that the sequence 
{ul{x),vl{x)) := {ui{x)-{ri)^U\:^),vi{x)-{rl^)^V\fr)) also satisfies 


( ll«,Oll|. = + 0(1), 

lA'iul.vD^Oin^*, (4.62) 

[a{uI,vI) ^ c - $((7°, UO) - A{U\V^). 

We also note that ([/^, V^) ^ (0, 0), 

A{U\V^)>Coo- 

By iterating the above procedure, we construct critical points (C/1, V^) of A and se¬ 
quences (rl) with the above properties, and, since $(u„, t;„) —?► c, the iteration must 
terminate after a finite number of steps. □ 


Remark 4.6. Recalling that 




■■= [ (|V<p + |V<p)dx 

Jn 


and 


{ul,vl)\\% 


|v<p + A*-^ + |Vu;p + p* , 

' x'^0 ' a;'"’ 


dx. 


we using the fact that for i > 1, ||(m)i,'u)j)||@ = ||(u)j, + o(l) in the proof above 

since that 


[ p^dx = o{l), [ \^dx = o{l) 

Jn Jn \x\^° 


by Lemma 3.6. 
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5 Variational identity and its applications 


For any (it, v) G we denote that 


a{u,v) :=\\{u,v)\\%, 


b{u,v) := 
c{u,v) := ^ 


(5.1) 


and define 


<^{u,v) := ^a{u,v) - “ nc{u,v) 


for all (it, v) G 3). 

Lemma 5.1. Any solution (it, v) of 


-Z\it -f A* 7 


- d '"' |I|n = 

(it, v) G 


/«n, 

in f2. 


satisfies 


1 


lV-2 


/ |V(it, i;)|^x • iidtT 
JdQ 


N - f 


7V-2 


/n FI' 


N-2 


In FI' 


-da; + A 


It 


2*(si) 


-dx 


t2*(^<l) 


+ M 


/n FI' 


-dx + 2 *(s 2 )k 


;|/5 


a Fr 

2 , 


/n FI' 


dx— |V(it, f)!^ dx. 


(5.2) 


In particular, iffl = M.f orM.^, then 


^ /■ |lt| 2 *(«i) /• |x| 2 *(^i) 

— A / —;——dx — p. I —:—-— 


Itr , N - rjo 


dx + 


N-2 


■d 


/n F 


Ito 


-dx 


dn FI 

Proof We take 

G(x, It, v) = 

2 IxFo 


/a Fr 


Jn 


(5.3) 


-Al 


|2*(^il) 


2"^ jxl'^o 2*(si) |x|®i 

A direct calculation shows that 


2*(si)' 


N 


Ex7G..(x,it,it)=^A*^ + fM 


m , * FI 


|2*(si) 


^|lt 


It m 




2*(si) 


i=l 


2 ' |x|‘^'> ' 2 jxl'^o 2*(si) |x|®i 

Si |it|“|xF 

p I I„. - KS2- 


2*(si) 


F r 


F ' 
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Then by [40, Proposition 7.1], we obtain (5.2) and (5.3). 
Corollary 5.1. Any solution {u, v) of 


-Au + - A 

-Av + 


[Tpi 


{u, v) e 




in 12, 
in 12, 


□ 


satisfies 

( |V(u,t;)pa; • i/dCT + (2 - cro)A* / + (2 - 770 )^* / 

JdQ Jq m ° Jq \xr° 

-(N-2){2*{s2)-a-P)KjJ-^^^^dx = 0. (5.4) 

Moreover, ifVl = orM.^, then 

-(iV-2)(2*(s2)-a-/3)At^^^^^dx = 0. (5.5) 

Proof. Since (u, v) is a solution, we have u), (u, v)) = 0. That is 

a{u, v) — b{u, v) — K{a + f3)c(u, v) = 0. (5.6) 


On the other hand, by (5.2), we have that 


a{u, v) — b{u, v) — K{a + P)c{u, v) 


2 — (To 


A* 


L 


^ J 2 - 770 ,, 
ux —7 ^—-fi 


N-2' 
\u\'"\v\^ 


/n 


dx 


N -2 

+ {2*{s2)-a-fi)K j dx-— ^ f \V{u,v)\^x-uda. (5.7) 

Jn Fl - z Jqq 


By (5.6) and (5.7), we obtain (5.4). When 12 = or R^, by (5.3) and (5.6), we 
obtain (5.5). □ 

Corollary 5.2. Iffl is a star-shaped domain o/R^ around 0, especially 12 = R:!J( or 
R'^, 0 < (To, 770, S 2 < 2, 0 < Si < 2, A* > 0, p* > 0, then (0, 0) is the unique solution 
of 

(-Au-\-X*jf^-X^ALAl^^ = Kaj^\u\°‘-^u\v\>^ inn, 

I -Au + p*-^ - m '"'' = Kl3j^\u\°^\v\P-^v in 12, 

[iu,v) e 

if furthermore (2* ( 52 ) — a — /3 )k < 0. 
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Proof. Since O is a star-shaped domain around 0, we have a; • > 0 on 9r2\{0}, where 

V denote the outward unit normal to dVl. Then by (5.4), we can obtain the result. And 
when fl = or by (5.5), we can also obtain the result. □ 

We prefer to give some this kind result about a single equation to close this section. 
The following results we refer to [18]. 

Proposition 5.1. (cf.[18, Proposition 2.1]) Let u G H^{n)\{0} be a solution of 
equation —Au = g{x,u) and G{x,u) = g{x,s)ds is such that G'(-,m(-)) and 
XiGxi (•, u(')) are in L^(0), then u satisfies: 


N 


idn 


\Vu\^x-gdSx = 27V 


G{x,u)dx+2'S2 / XiGxi{x,u)dx—{N—2) / jVupdx, 
! Jn Jn 


where Lt is a regular domain in and g denotes the unitary exterior normal vector 
to dQ. Moreover, ifLl = then 


N 

27V f G{x,u)dx + 2^^ ( XiGxi{x,u)dx = {N — 2) f \\/u\‘^dx. 

^ Jr" 7rn 


Remark 5.1. Since the processes are standard now, we give the following results with¬ 
out proof: 

Let LI = and u G £)q’^(O)\{0} be a solution of equation —Au = g{x,u) and 

G{x, u) = g{x, s)ds is such that G(-, u{- )) and XiGxi[-,u{-)') are in L^{Ll), then 
u satisfies 


N 

27V [ G{x,u)dx2^^ [ XiGxi{x,u)dx = {N — 2) [ \S/u\'^dx. 
jRf: ^ JRf JRf 


□ 


Corollary 5.3. Let Ll = or R-^, then problem 
_a„ + e E := D«(n) n L-+' fn,^) . (5.8) 

has no nontrivial solution if X f 0 andp f 2*(si) — 1. 

Proof. Let g{x, u) = —- and u be a solution of (5.8). A direct 


calculation shows that 

N 


Tx.Gxfx u)- 

■“ » ’ P+1 |a;|®i 2 *(s 2) |a;|®2 


i=l 


Then by Remark5.1 and Proposition5.1, we have 


f iVitpdx — f 
Jn Jn 


2*( s 2) -2*(si)A f \u\P+^ 

-dx = ' 


p + 1 


dx. 


(5.9) 
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On the other hand, tested by u, we have 



(5.10) 


Hence, if A ^ 0 and p ^ 2* (si) — 1, tt = 0 is the unique solution. 


□ 


6 Estimation on the least energy mg 

Remark 6.1. Under the assumptions of this section, it is easy to prove the correspond¬ 
ing Nehari manifold Jf is well defined, and Af is closed and bounded away from (0, 0). 
We can also prove that any {PS)c sequence o/$ is bounded in And any bounded 
PS sequence o/$|^ is also a bounded PS sequence o/$. Since these proofs are very 
standard, we omit the details and refer to [40, Section 4]. 

Remark 6.2. We always assume that (Hi) and {H 2 ) are satisfied, especially when 
|f2| = 00 , we assume further that (^^ 0 ) (^ 170 ) hold(see subsection 4.2), which 



We can obtain the following results which are parallel to those in [40, subsection 6.2[ 
and [41, subsection 2.2[, see also [1, 36, 37[.etc. Although the framework spaces are 
different, the arguments depends only on the values of a and /3, these proofs can be 
modified smoothly. Since the proofs are long but standard, we omit the details. 

Lemma 6.1. Assume (i? 2 ), 1 < ct, 1 < /3, a + /3 < 2*(s2), and if \il\ = 
00 , a + /3 < 2* ( 52 ), we assume further that (^^o) (^> 70 ) 0 < U € E is 

a least energy solution of (1.9) and define 



( 6 . 1 ) 


Then 


(1) if K < 0 or f > 2 or < , {U,0) is a local minimal of $ in M. 

[k < ?7i 

(2) if (3 < 2, K > 0 or /3 = 2, K > 7 ) 1 , 

mo := inf ^>(0, (/?) < ^{U,0) = \IJJ) = 0 ^ 0 .A*,A- (6.2) 


Proof. We omit it. 


□ 


Lemma 6.2. Assume {Hi),{H 2 ),l < a, 1 < /3, a + /3 < 2*(s2), and if \V,\ = 
00 , a + /3 < 2* ( 52 ). we assume further that and (A*^) hold. Let 0 <V G F is 
a least energy solution 0 / (1.11) and define 



(6.3) 


Then 
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, (0, V) is a local minimal o/$ in Af. 


(1) if K<0 or a>2 or < 

[K <fl 2 

(2) if a < 2, K > 0 or a = 2, K > 7)2, 

mo := inf $(<?!), </?)< $(0, F) = T^(y) = (6.4) 

(0,V)G7V 

Proof We omit it. □ 

Corollary 6.1. Assume (iJ 2 ), 1 < ct, 1 < /3, q: + /3 < 2*{s2), and if \^l\ = 
oo, a + /3 < 2 *(s2 ). we assume further that {A*^^) and (A*^) hold. If (3 < 2 or (3 = 2 
with K > fji, and if a < 2 or a = 2 with k > 7)2, then we have 

mo < min{co.(,,A*,A,Cr,o,;i.,^}. (6.5) 

Proof By Lemma 6.1 and Lemma 6.2. □ 

7 Proofs of Theorem 1.4 and Theorem 1.5 

Proof of Theorem 1.4: We note that under the assumptions, it is easy to see that the 
corresponding Nehari manifold is well defined. Let {{un, Vn)} C Af he a minimizing 
sequence, then it is standard to prove that {{un,Vn)} is a bounded {PS)mo sequence 
for $. And by Corollary 6.1, we have 


mo < min{co.o,A*.A,c,,o.A‘*.M}- (^-l) 

Hence, by Corollary 4.4, we obtain that there exists some {u, v) € & such that (u„, u„) —>■ 
{u, v) and u ^ 0, u ^ 0. Thus, (u, v) is a nontrivial ground state solution. □ 


For the case of a + (3 = 2*(s2), it is not easy to see that Un —?> u or 
dx , 

(see Remark 4.4). Hence, for the critical couple case. 


strongly in j 


we can note easily obtain the compactness result basing on Corollary 6.1 and Corollary 
4.4. We also note that 


min|^C(7o , A* , A ; Oiyg ^ 


2 *(si)J 




2*(«l) 

2*(ni)-2 


(max{A, /r}) 


2*(<'2)-2 


(7.2) 

We let 0 denote the least energy of the limit problem, then by [40, Theorem 7.2] or 
[41, Theorem 1.1], we also have 


0 < 


1 

2 


1 

2^_ 


" (max{A,/r}) 


2 

2*(s2)-2 _ 


(7.3) 


Hence, we can not judge the relationship of size between min{ccro,A*,A, 

0 intuitively. To apply the splitting Theorem 4.2, we need a further detailed estimation. 
For the technical reasons, we need the assumptions {Hi) and (772)- 
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Proof of Theorem 1.5: Similar to the proof of Theorem 1.4, there exists a minimizing 
sequence {(un, t'n)} C Af which is a bounded {PS)mo sequence of <i>. Moreover, 

mo < min{ccro,A*.A,c^o,A.*,/x}- (7-4) 

Under the assumptions of {Hi) and {H 2 ), we shall prove that mo < 0 and thus 

mo < mm{0, (7.5) 


Then, applying the splitting Theorem 4.2, we see that there exists some u 0,v ^ 0 
such that {un, Vn) —>■ {u, v). Hence, {u, v) is a nontrivial ground state solution. 

The proof of mo < 0 under the assumptions (Hi) and (f/ 2 ): Denote the functional 
corresponding to the limit equation by A, under our assumptions, by [40, Theorem 
7.2] or [41, Theorem 1.1 ], we see that the limite equation possesses a nontrivial ground 
state solution, i.e., there exists some u ^ 0,v ^ 0 such that A{u,v) = 0 and {u,v) 
satisfies the following equation: 


|„|2 («2)-2„ 

K > 0, (it, v) G V := 


= naj^\ur-^u\vf 
= Kl3j^\u\’^\v\^-^v 
dI'^{R^) X dI'^{R^) 


in 

in R^, 


(7.6) 


Without any loss of generality, we may assume that, in a suitable neighborhood of 
0 , dfl can be represented as xn = <f{x'), where x' = (ii,--- ,a:Ar_i), <p(0 ) = 
0, VV(0) = 0, V' = - jiOat-i), and that the outer normal to OH at 0 is 

—e-N = (0, • • • I Oj ~1)- Define (j){x) = {x',X m — '^{x')) to “flatten out” the boundary. 
We can choose a small xq > 0 and neighborhoods of 0, C/ and [/, such that ((>([/) = 
]Bro(0), 4'{U n H) = B[!'(0), (/)([/) = Bio(O) and(/)(17 n H) = B^(0), where, for 

2 2 

any ro > 0, Bro(O) C is an open ball of radius xq centered at 0 and B+(0) = 
Brp (0) n By the smoothness assumption on OH, (p can be written as 


AT-l 

‘pi.y') = (7-7) 

2=1 


then 


For any £ > 0, define 


N-l 


"(») = E «■ 

2=1 


/ N f Hx)\ I s 

Ue{x) := £ 2 u - I ^Ve{x) := £ 2 ^ ' 


a; £ H n [/, 


and let 


,{x) = r]Ue{x),Ve{x) = T]Ve{x). 


(7.8) 

(7.9) 
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For the estimation on the gradient terms, similar to [39, Lemma 3.4], we have 

[ \'^Ue{x)f dx = f \Vu{y)f dy-2 f rj {(l)~^{ey)y dNu{y)V'u{y)iV'ip){ey')dy+0{e‘^ 
JQ. JK.^ JbL 


(7.10) 


and 


[ \Vve{x)\^ dx = [ \Vv{y)f dy-2 [ y (cj) ^{ey)Y d^v{y)Wv{y)-{V'(p){ey')dy+0{£'^) 
Jn 7b+q 

(7.11) 

Using the decay property of u and v (see [40, Theorem 7.2] or [41, Theorem 1.1]), by 
(7.7) we have (the details we refer to [39, Lemma 3.4]): 

-2 f y[(j)~^{ey)f dNu{y)Vu{y) ■ {SI'ip){£y')dy 

e 

o p N-1 

V (<^"^(ey))^ dNu{y) ^ d^,u{y)ip{£y')dy + O(e^) 


£ 7b7o 


(7.12) 




and 


-2 f y{(j) ^{£y))'^ dNv{y)Vv{y) ■ {V'(p){£y')dy 
Jata 

e 

r, /* ^-1 

y {(j)~^{£y)f' dNv{y) ^ duv{y)<f{£y')dy + O(e^) 


£ 7b^ 


(7.13) 


2=1 
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By equation (7.6) and the formula of integration by parts, we have 


N-l 


£ ./B + 


\ey)) dNuiy) diiu{y)y}{ey')dy 


£ 


7 ^{£y)T ^Nu{y) [Au{y) - dNNu{y)] ip{ey')dy 




£ 

e 

2 A 


y{(j) ^{ey))'^ dNu{y) 


' 1 u{y)° ^v{yY 

A-r-;-h Ka- 


\y\‘ 


\y\‘ 


£ 2*(si) Jk+ 


y{4' ^{£y)Y dN[dNu{y)f (p{ey')dy 


- -At / y{(j) ^{ey)) 

^ 7b+ 


2 On {u{y)°‘)v{yY 


(p{ey')dy 


£ -/B^naR^ 


Similarly, we have 
2 


77 ((/)"^(ey))^ {dNu{y)f (p{ey')dSy + O(e^) 


N-l 


v{4> \£y)) dNv{y)J2 diiv{y)ip{ey')dy 
2 y 


£ 


i=l 


£ 2*(si) 7 b+ 


?? (<^ (£?/)) --‘/^>(£ 2 / )dy 






77 (cj) ^{ey))^ {dNv{yN ‘f{£y')dSy + 0{e^) 


if{ey')dy 


(7.14) 


(7.15) 
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Then, by (7.10)-(7.15) and the formula of integration by parts, we have 

I \VUe{x)\'^dx + I \VVe{x)\'^dx 

Jn Jn 


in 

\Vu{y)\'^dy+ [ \Vv{y)\'^dy 

N ImN 

2 1 


A-if 


^ 2*(si) J^+ 


ro 




“J“L ’’(* K ley')<ly 




7(</> ^(ey)) idNu{y)) + idNv{y)) ip{ey')dSy + 0{e^) 


/Ri^ 


\Vu{y)\'^dy+ / \Vv{y)\'^dy 


/R" 


2 1 


e 2*(si) J^+ 


v{^ {ey)) --¥^(ey)rf2/ 


2 f ^^2 S2{u{y)°‘v{yf)yN , 

" e"' _4+ ^^y 

e 

+ - [ y kdNuiy)f + {dNv{y)f] (p{ey')dSy + O(e^) 

e 

:= [ \\/u{y)\'^dy+ [ \Vv{y)\‘^dy + Ji + J 2 + J 3 + 0{e'^). (7.16) 

7rw Jrw 


where 


Ji — — 


e 

2si f (Au(y)^*^'*'^ yw, ,| 2 j 2 ^ 

' —^-|y rdyi7(0) (1+ o(l))e + 0(£:'‘) 


2 *(si) Jr^ 

2si 


\y\S,+2 


2 *isi) 


KiH{Q) {1 + o{l)) e + 0{e^), 


(7.17) 


J2=--ks2 [ p((/) ^(ey))^- 

£ 7b1:„ 


u{y)°‘v{y)^) yN 


<f{ey')dy 


= — 2kS2 


|y|S2+2 

_li 

e 

f {u{y)°‘v{y)^) yN. ,,2, 2^ 

^- \y\s2+2 - \y I dyH{0) (1 + o(l)) e + 0{e ) 


:= - 2 ks 2 K 2 H{ 0 ) (1 + o(l)) e + 0{e^), 


(7.18) 
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Jz =- 


v{4‘ \£y)) {dNuiy)) +{dNv{y)) ip{ey')dS, 


e 

\{dMu){y’, {))f + {{dMv){y'. 0))"1 |yf dy'iJ(O) (1 + o(l)) £ + 0{e^) 

r-l L J 

:=i^3H(0)(l + o(l))e + O(e2). (7.19) 

On the other hand, by the standard estimation, we have 


-dx = 


In Fl 


R« \yr Jr^ I2/I + ' 

(7.20) 




|z;(y)p*(*0 




n FI 


dx= I - dy- I ' """ (1 + o(l))g+0(£^)) 




/ 


|Me(a^)P 

O-Q 


a F 


\Ve{x)\^ 


Jn 

\M^rMx)f 


dx = 


dx = 


m’l 


Hy)? 


\y\ 


(7.21) 


'Kif I 2 /I 


<^0 


dyil + o{l))+0{e^), (7.22) 


'''^^^'"-d?/(l + o(l)) + 0(e^), (7.23) 


*R" I 2 /I 




/n Fr 




f S2\u{y)\°‘\v{y)\l^\y'\^yN 


(7.24) 


Then, basing on the sequence of estimations above, we have 


<- 




[ \Vu{y)\'^dy+ f \Vv{y)\'^dy+ (K 3 - Ki - 2ks 2^2 ) 77(0) (1 + o(l))£ 

R^ 7rn V 2 *(si) 


2 

L“ 

+ A * / (1 + 0 ( 1 )) £ 2-0 + 


f W{y)\^ 
Ir^ I 2 /I'"” 


k(2/)P 


^2*(si) 

"24^4 

- Kf2*(s2) 


A|m(?/)|2*('>i) +^|^(y)|2*(si) 


I 2 /I" 


dy — si.^iid(0) (1 + 0 ( 1 )) e + O(e^) 


m u 


1^ 


Ir^ li/h 


dy - S 2 K 2 H{ 0 ) (1 + 0 ( 1 )) £ + 0(£2) 


(7.25) 


and it follows easily that there exists some T > 0 large enough and some £0 > 0 small 
enough such that 

<I)(Tu£, Tug) < 0 for all 0 < £ < £ 0 . (7.26) 

Especially, under the assumptions of(i7i and {H 2 ), we can apply the similar arguments 
as [39, Lemma 3.4] and obtain that there exists some 0 < £1 < £0 such that 

max<i)(t?},e, fue) < max A(tu, tv) = 0 for all 0 < £ < £ 1 . (7.27) 

t>o t>o 
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